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1.  INTRODUCTION 


This  report  provides  a  detailed  study  of  the  design  of  feasible  human-robot  interfaces  for 
near-term  deployment  in  a  “robot  unit,”  defined  as  a  tightly  coupled  group  of  humans  using  a 
multiplicity  of  robots  as  tactical  tools.  There  is  a  strong  relationship  between  three  phases  of 
fielding  man-machine  systems  of  this  type:  system  design,  operator  selection,  and  operator  train¬ 
ing  (Figure  1).  Here  we  consider  all  of  these  dimensions,  developing  an  understanding  of 

•  the  tradeoffs  between  highly-trained  operators  versus  novice  operators, 

•  the  importance  of  specific  cognitive  and  intellectual  reasoning  abilities  of  potential 
operators,  and 

•  the  impact  of  system  design  on  all  of  this. 

Clearly,  a  sophisticated,  well-designed  system  will  require  less  training  and  enable  a  larger 
set  of  people  to  interact  with  it.  The  purpose  of  this  study  is  to  span  this  space  of  potential  design 
and  human  factors  issues  and  identify  the  inherent  wins,  losses,  and  trade-offs  given  the  goal  of 
rapidly  fielding  such  a  system. 


Figure  1:  The  implementation  of  a  usable  man-machine  system  requires  a  synergistic 
approach  that  takes  into  account  both  the  hardware  and  the  operator. 

To  accomplish  this,  we  look  heavily  towards  existing  literature  and  studies.  Thus  we  attempt 
to  synthesize  information  from  a  wide  range  of  disparate  sources  including  published  papers, 
military  documents,  and  commercial  sources.  Visits  have  been  taken  to  sites  that  are  promising 
nuclei  of  relevant  information,  including: 
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•  Oak  Ridge  National  Laboratory  (telepresence  concepts,  teleoperation  techniques, 
and  human  factors), 

•  NASA  Ames  (human  factors  techniques),  and 

•  Ft.  Benning. 

A  guiding  influence  of  this  work  is  the  report  of  the  TMR  Concept  Development  Group, 
“Concept  Development  Report  For  The  Tactical  Mobile  Robotics  (TMR)  Program  (Extract  of 
03/24/1999  Revision),”  which  is  included  as  Appendix  A  of  this  document  and  will  be  referred 
to  often  [TMR  99] .  Often,  the  CDG  recommendations  serve  to  constrain  the  problem  and  focus 
this  study,  and  in  many  cases  we  cite  evidence  of  the  validity  of  the  CDG  opinions.  In  a  few 
cases,  we  identify  areas  where  some  promising  technologies  may  have  escaped  the  attention  of 
the  CDG. 

2.  HUMAN-ROBOT  INTERFACE  DESIGN 


This  section  is  intended  to  serve  military  decision  makers  as  a  guide  to  existing  technologies 
for  human-robot  interfaces  that  are  practical  for  field  deployment.  In  this  context,  “practical” 
means  that  the  technology  has  been  demonstrated  to  be  reliable,  somewhat  ruggedized,  and  ide¬ 
ally  available  as  commercial  off-the-shelf  (COTS). 

The  “system”  being  considered  is  the  combined  system  of  a  human,  one  or  more  robots,  and 
appropriate  interface  hardware  and  software.  All  robots  have  readily-identifiable  weaknesses,  as 
do  humans.  Therefore,  a  major  point  of  this  report  is  to  assess  the  strengths  and  weaknesses  of  a 
human  augmented  with  the  remote  resources  of  a  robot.  In  order  to  justify  the  use  of  robots,  is 
not  sufficient  to  merely  to  note  that  a  robot  can  have  specialized  “non-human”  sensors  or  loco¬ 
motive  capabilities.  It  must  be  shown  that  there  are  reasonable  methods  to  allow  a  human  opera¬ 
tor  to  have  access  through  these  capabilities  through  available  technology  while  still  maintaining 
awareness  of  local  surroundings. 

While  interfaces  for  teleoperated  and  teleautonomous  control  are  getting  more  attention  and 
becoming  increasingly  more  sophisticated,  the  primary  emphasis  of  existing  systems  is  concen¬ 
trated  on  non-man-portable  systems,  such  as  multi-operator  control  stations  for  high-value  UAVs 
and  elaborate  mission  control  centers  for  space  robotics.  The  field  is  considerably  narrowed 
when  dealing  with  the  targeted  application  of  an  single  dismounted  soldier  controlling  multiple 
platforms. 

2.1  Wearable  Computing  and  Augmented  Reality 

A  complete  robot  OCU  suitable  for  field  deployment  in  tactical  situations  has  much  in 
common  with  what  is  generally  called  a  “wearable  computer.”  To  emphasize  just  how  closely 
the  model  of  wearable  computing  fits  the  TMR  requirement,  consider  the  criteria  given  by  Mann 
as  a  basic  definition  of  the  subject  [Mann  97].  Here  he  states  that  a  wearable  computer  is: 

“1.  eudaemonic:  the  apparatus  is  subsumed  into  the  ‘eudaemonic  space’  of  the  wearer  (e.g. 
it  may  be  worn,  or  otherwise  situated  in  a  manner  that  makes  it  part  of  what  the  user  considers 
‘himself  or  ‘herself,’  and  in  a  manner  that  others  also  regard  it  as  part  of  the  user  .  .  .).  It  is  suffi¬ 
cient  that  the  interface  be  eudaemonic  (e.g.  some  of  the  compute  power  can  be  remote); 
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“2.  existential:  the  apparatus  is  controlled  by  the  wearer.  This  control  need  not  require  con¬ 
scious  thought,  but  the  locus  of  control  must  be  such  that  it  is  entirely  within  the  wearer’s  domain 
....  Furthermore,  the  apparatus  provides  the  wearer  with  the  ability  to  make  its  operation  com¬ 
pletely  private  and  secure  when  desired.  In  addition  to  the  obvious  privacy  afforded  by  its  eu- 
daemonic  nature  (e.g.  securely  attached  to  the  body  so  that  it  might  be  less  likely  to  be  stolen 
when  working  in  close  quarters  with  adversaries),  the  output  can  be  made  private  when  desired 
by,  for  example,  using  a  screen  that  cannot  be  read  by  others  looking  over  the  wearer’s  shoulder; 

“3.  constant:  (constancy  of  both  operation  and  interaction): 

•  Operational  constancy:  it  is  always  active  while  worn. . . . 

•  Interactional  constancy:  one  or  more  output  channels  (e.g.  screen/viewfinder)  are 
known  (e.g.  visible)  to  the  user  at  all  times,  not  just  when  the  computer  is  being  in¬ 
teracted  with  in  a  deliberate  manner.  In  this  way,  context  switching  time  to  mentally 
engage  with  the  apparatus  is  near  zero.” 

While  Mann  generally  considers  the  above  criteria  in  the  context  of  a  civilian  user,  enhanc¬ 
ing  the  security  and  freedom  of  the  individual,  the  applicability  to  a  military  user  is  clear.  In 
agreement  with  the  principles  described  in  Section  6.2  of  the  TMR  CDG  [TMR  99]  (“Human 
Robot  Interface”),  Mann  also  states  [Mann  98]  that  essential  qualities  of  a  wearable  computer 
include  that  it  be 

1.  unmonopolizing  of  the  user’s  attention, 

2.  unrestrictive  to  the  user, 

3.  observable  by  the  user:  “It  can  get  your  attention  continuously  if  you  want  it  to,” 

4.  controllable  by  the  user:  (“Responsive”), 

5.  attentive  to  the  environment,  and 

6.  communicative  to  others. 

The  single  greatest  difference  between  a  generic  wearable  computer  and  a  TMR  OCU  is  that 
the  OCU  is  specialized  to  act  as  a  remote  teleoperation  “console,”  and  as  such  may  require  spe¬ 
cialized  input  devices  for  robot  control  and  specialized  output  devices  for  robot  sensor  visualiza¬ 
tion.  In  a  word,  a  TMR  OCU  requires  some  degree  of  teleimmersion  or  telexistence,  a  computer 
user’s  experience  of  being  part  of  a  remote  environment.  Wearable  computers  developed  to  date 
sometimes  include  some  capabilities  related  to  either  teleimmersion  or  ordinary  immersion  (a 
virtual  environment  rather  than  a  remote  environment),  but  this  is  essentially  an  application- 
dependent  characteristic. 

We  must  be  careful  to  define  our  concept  of  teleimmersion  in  the  OCU  context,  since  im¬ 
mersion  (and  similarly,  teleimmersion)  can  imply  a  monopolizing  user  interface.  At  the  extreme 
end  of  the  teleimmersion  scale  is  virtual  reality,  a  poor  model  for  what  a  TMR  operator  should 
experience,  since  by  definition  it  is  overwhelmingly  distractive.  But  it  can  be  useful  for  an  op¬ 
erator  to  at  will  step  in  and  out  of  an  environment  that  embodies  some  of  the  qualities  of  aug¬ 
mented  reality  and  mediated  reality.  (Later  we  discuss  some  of  the  reasons  for  this,  including 
improved  ability  to  perform  remote  driving  and  manipulation  operations.) 
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Augmented  reality  refers  to  the  enhancement  of  the  actual  perceived  environment  with  in¬ 
formation  that  has  been  obtained  by  other  means.  A  heads-up  display  in  an  aircraft  or  other  vehi¬ 
cle  is  a  common  example,  where  graphical  and  textual  information  about  location,  altitude,  and 
vehicle  status  are  made  available  while  still  allowing  the  pilot  to  see  the  local  surroundings.  Any 
soldier  carrying  wireless  voice  communications  equipment  is  essentially  experiencing  a  basic 
form  of  audio-augmented  reality,  enhancing  the  visual  experiences  of  the  local  environment  with 
useful  information  from  other  military  personnel  (and  generally  augmenting  their  “reality”  with 
return  communication,  as  well). 

Mediated  reality  acts  as  a  filter  on  the  perceived  environment,  blocking  extraneous  informa¬ 
tion  and  focusing  user  attention  on  important  stimuli.  In  the  TMR  OCU  scenario,  we  can  imag¬ 
ine  multiple  operators  in  close  proximity  to  each  other  and  a  support  HMMWV,  all  potentially 
exposed  to  enemy  fire.  So  an  approach  based  on  mediated  reality  would  attempt  to  block  out 
some  of  the  activity  of  the  other  operators  and  support  personnel  while  focusing  operator  atten¬ 
tion  on  the  following  important  information: 

•  status  of  the  robots  under  the  operator’s  control, 

•  warnings  and  other  pertinent  information  from  sentries  providing  protective  support 
for  the  robot  unit, 

•  high-level  status  of  robot  teams  under  control  of  other  operators,  and 

•  operator’s  own  perceptions  of  local  threats  (sniper  fire,  aircraft  activity,  etc.). 

This  is  a  key  point:  mediated  reality  should  be  better  than  reality.  Instead  of  concerning 
ourselves  with  how  an  OCU  may  impair  the  operator’s  ability  to  deal  with  other  “non-robot- 
related”  activities,  we  must  turn  the  problem  back  on  itself  and  strive  make  the  operator’s  situ¬ 
ational  awareness  better  than  it  would  be  without  the  OCU. 

Wearable  computing  is  often  considered  to  be  a  subset  of  ubiquitous  computing,  the  notion 
of  having  computers  literally  everywhere,  in  even  the  simplest  devices.  Curiously,  some  con¬ 
sider  ubiquitous  computing  to  be  the  opposite  of  virtual  reality,  by  virtue  of  the  fact  that  placing 
computers  in  a  user’s  world  is  the  opposite  of  placing  the  user  in  a  computer’s  world.  But  clearly 
that  is  a  limited  viewpoint  -  the  worlds  can  mix  in  almost  any  proportion,  and  a  TMR  OCU  rep¬ 
resents  such  a  mix. 

Most  existing  COTS  wearable  technology  does  not  address  the  teleoperation  issues,  instead 
focusing  primarily  on  data  entry  and  database  access  applications  including  inventory,  mainte¬ 
nance,  inspection,  medical  treatment,  cartography,  and  journalism  (Figure  2).  Some  of  these  ap¬ 
plications  are  described  at  the  web  site  of  a  leading  vendor  of  wearable  computers,  Xybernaut 

(www.xvbemaut.com). 
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Figure  2:  Emerging  applications  of  COTS  wearable  computers,  including  (left  to  right)  car¬ 
tography,  journalism,  and  inspection/maintenance.  (  ©  Copyright  1999  Xybernaut  Corpora¬ 
tion  and  subject  to  use  restrictions  at  http;//www.xybernaut.com.) 


Of  slightly  greater  interest  are  applications  related  to  “location-aware”  access  of  data,  such 
as  those  projects  pursued  by  the  Nexus  group  at  the  University  of  Stuttgart  (http://inf.infoniiatik.uni- 
stuttgart.de/ipvr/vs/proiekte/nexus/).  the  Futurc  Computing  Environments  and  Wearables  groups  at  Georgia 

Tech  (http://www.cc.gatech.edu/gvu/fce/index.html.  httr)://wearables. gatech.edu).  thc  W^Carable  Computing  and  Vision 

and  Modeling  groups  at  MIT  (http://www-white.media.mit.edu/vismod). 

http://wearabies.www.media.mit.edu/proiects/wearahies).  the  Wearable  Computing  Systcms  group  at  Camcgic 
Mellon  (http://www.cs.cmu.edu/afs/cs/proiect/vuman/www/frontpage.htmi).  and  othcrs.  Location  awarcncss  Can  bc  on 

a  large  scale,  as  in  aids  for  a  tourist  exploring  an  unknown  city  with  the  aid  of  a  wearable  com¬ 
puter  that  displays  information  relative  to  GPS  position  (Figure  3a),  or  on  a  local  scale,  as  in  the 
use  of  a  pose  estimation  sensor  (e.g.,  a  head  tracker)  to  determine  where  a  user  is  looking  so  that 
his  view  may  be  annotated  (Figure  3b).  Both  of  these  are  examples  of  augmented  reality,  and 
both  may  be  relevant  to  a  TMR  OCU. 
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Figure  3:  Applications  of  location  awareness  for  wearable  computers,  a)  A  tourist  access¬ 
ing  location  dependent  data  in  an  unknown  city,  and  b)  Machinery  annotated  with  informa¬ 
tion  relative  to  the  user’s  view.  (  ©  Copyright  1999  Xybernaut  Corporation  and  subject  to 
use  restrictions  at  http://www.xybernaut.com.) 


Ideally,  the  core  elements  of  a  TMR  OCU  would  be  interchangeable  with  any  standard-issue 
wearable  computer  which  is  ultimately  developed  for  military  use.  Such  concepts  typically  in¬ 
clude  a  computer,  power  supply,  voice  radio,  GPS,  display,  and  input  devices.  As  will  be  shown, 
the  major  differences  are  likely  to  be  in  the  display  and  input  devices. 

In  some  of  the  more  relevant  research  involving  wearable  computers,  Barfield  [Barfield98] 
considered  the  effectiveness  of  augmented  reality  (AR)  displays  in  aiding  an  operator  in  the  per¬ 
formance  of  a  manufacturing  assembly  task.  The  researchers  compared  four  methods  of  convey¬ 
ing  the  assembly  instructions  to  the  operator: 

•  printed  instructions, 

•  conventional  computer-aided  instructions, 

•  information  graphics  on  an  opaque  AR  display,  and 

•  information  graphics  on  a  transparent  AR  display. 

In  the  Barfield  study,  the  test  subjects  were  given  the  task  of  assembling  a  computer  mother¬ 
board,  admittedly  a  significant  extrapolation  of  a  typical  TMR  task.  They  were  given  training  on 
the  insertion  process  for  various  components  prior  to  undergoing  trials  in  which  the  metrics 
measured  were  1)  time  of  assembly  and  2)  number  of  errors.  This  was  followed  by  a  question¬ 
naire  to  assess  user  preferences.  The  final  results  showed  lowest  execution  times  for  the  trans¬ 
parent  AR  display,  followed  by  the  opaque  AR  display,  computer-aided  instructions,  and  printed 
instructions.  Errors  were  lower  with  AR  displays  in  general.  Certainly  this  supports  the  notion 
of  using  AR  technology  to  facilitate  the  operation  of  a  TMR  OCU,  but  it  does  not  address  the  ef¬ 
fectiveness  of  wearable  technology  for  teleoperation.  Another  Barfield  study  briefly  described  in 
the  same  report  supported  the  use  of  force  feedback  in  the  understanding  of  statics  and  dynamics. 

The  usefulness  of  AR  as  an  effective  model  for  providing  information  to  the  warfighter  has 
not  been  unnoticed  by  the  military.  As  noted  in  COTS  Journal  [Ciufo  00]: 
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.  the  trouble  with  immersive  HMDs  is  that  they  block  out  the  real  world  and 
prevent  the  operator  from  reacting  to  real  events  while  immersed  in  the  virtual  world. 
Whereas  this  downside  can  be  overcome  by  piping  in  real-world  video  on  top  of  the  vir¬ 
tual  environment,  information  overload  sets  in,  and  the  operator  can  quickly  become 
disoriented.  A  better  approach  is  augmented  reality  (AR)  technologies  that  allow  view¬ 
ing  the  real  world  with  superimposed  virtual  objects. 

“The  U.S.  military  is  using  AR  in  a  big  way.  HMDs  are  used  by  pilots  to  supple¬ 
ment  the  traditional  heads-up  display  (HUD)  in  platforms  ranging  from  the  F-15  and 
F/A-18  to  the  new  Joint  Strike  Fighter  and  Comanche  helicopter.  Armored  vehicle  driv¬ 
ers  and  commanders  will  use  HMDs  with  "head-out"  views  of  the  real  world  while  still 
viewing  vehicle  instruments  and  weapons  systems.  Battlefield  soldiers  have  a  digital 
view  of  the  battlefield,  the  locations  of  ‘friendlies’  and  opposing  forces  (OPFOR),  and 
remote  viewing  capabilities  around  comers  or  in  anti-laser  ‘eye-safe’  mode.  And  both 
medics  and  mechanics  can  call  up  on-the-spot  documentation  for  ready  access  to  on-site 
documentation.” 

A  key  issue  in  the  near-term  deployment  of  robot  units  is  the  ruggedness  of  available  equip¬ 
ment.  This  report  deals  primarily  with  COTS  equipment  and  some  devices  that  are  only  slightly 
past  the  proof-of-concept  stage.  In  that  respect,  mggedness  is  evaluated  in  terms  of  the  potential 
for  ruggedization.  This  results  in  three  possible  classifications  for  a  component  or  system:  1) 
ruggedized  (suitable  for  military  use  as  is),  2)  ruggedizable  (a  clear  path  for  ruggedization  exists, 
using  mature  technology  at  reasonable  cost),  or  3)  questionable  (fragility  exists,  and  there  is  no 
obvious  solution  at  this  time).  “Questionable”  technologies  will  not  be  utilized  in  the  proposed 
robot  units. 

It  is  probable  that  TMR  operators  will  often  perform  their  supervisory  tasks  within  close 
proximity  to  a  support  vehicle.  In  such  cases,  it  is  still  important  that  the  operator  remain  unen¬ 
cumbered  and  able  to  react  to  the  local  situation,  but  it  opens  up  possibilities  for  non-wearable 
devices  that  can  remain  resident  on  the  support  vehicle.  These  devices  will  be  considered  here, 
with  appropriate  notation  of  their  limitations  for  man-portable  applications. 

2.2  Input  Device  Technolosies 

This  section  describes  a  wide  range  of  input  device  technologies  in  the  context  of  a  TMR 
OCU.  The  evaluation  of  these  devices  with  regard  to  wearable  computers,  teleoperation,  or 
teleimmersion  has  resulted  mainly  from  the  experiences  at  major  centers  of  advanced  users. 

2.2.1  Pose  Estimation  Sensors 

Pose  estimation  sensors  are  devices  that  measure  some  relationship  of  one  or  more  of  an  op¬ 
erator’s  appendages  to  the  operator  or  a  reference  frame.  Typically,  this  relates  to  the  operators 
head,  hands,  or  fingers,  but  in  some  of  the  more  immersive  versions  it  may  include  virtually  the 
entire  body.  In  general,  the  pose  estimation  sensors  used  for  research  in  immersive  environments 
are  too  cumbersome  to  be  practical  for  a  wearable  TMR  OCU.  But  some  feasible  alternatives 
exist,  including  head  position  sensors  and  gloves,  which  will  be  described  in  the  following  sec¬ 
tions. 
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Pose  estimation  sensors  tend  to  be  analog  in  their  response  characteristics,  providing  some 
degree  of  continuous  control.  In  most  of  the  available  literature  on  robotic  teleoperation,  there  is 
little  mention  of  using  discrete  “cursor-key”  type  control.  The  MPRS  program  demonstrated 
some  effective  operation  of  teleoperated  (not  teleautonomous)  robots  with  keys  on  a  pendant  de¬ 
vice  [Laird  00],  but  it  requires  the  dedicated  use  of  both  hands.  This  sort  of  operation,  including 
the  adjustment  of  speed  with  a  rotary  potentiometer,  can  be  suitable  for  dedicated  control  of  a 
single  machine,  but  is  less  desirable  for  TMR. 

2.2.1. 1  Gloves 

While  human  operators  are  adaptable  to  a  variety  of  devices  using  different  types  of  muscu¬ 
lar  action,  the  need  for  continuous  control  appears  to  be  obvious.  The  CDG  advocates  the  use  of 
gloves  for  sending  commands.  In  this  context,  it  is  possible  to  consider  discrete  commands 
(through  the  recognition  of  specific  gestures)  or  continuous  control  (through  the  recognition  of 
degrees  of  finger  extension,  for  example).  In  the  survey  of  hardware  included  here,  no  considera¬ 
tion  is  given  to  gloves  which  are  part  of  large  suits  or  systems  which  are  clearly  impractical  for 
TMR,  nor  to  “toy”  devices  that  are  not  comparable  in  performance  and  durability. 

Pioneering  virtual  reality  research  was  performed  with  the  DataGlove  from  VPL  Research, 
whose  technology  has  since  been  purchased  by  Sun  Microsystems.  The  fiber  optic  sensors  used 
in  these  gloves  were  reported  to  be  fragile  and  subject  to  calibration  problems,  and  they  are  no 
longer  available.  Sun  is  apparently  more  interested  in  the  virtual  reality  software  developed  by 
VPL  Research  than  their  hardware,  and  none  of  the  original  VPL  devices  are  in  production.  But 
back  in  1987  when  they  first  produced  the  DataGlove,  VPL  Research  licensed  Nissho  Electronics 
as  their  sole  distributor  and  technical  partner  in  Japan.  Nissho  apparently  still  produces  their  ver¬ 
sion,  the  SuperGlove  (http://www.tradepia.or.ip/nevc/advanced/vr/vr5-ie.htm).  but  there  is  little  reported  rcscarch 
activity  with  this  device.  Supposedly,  some  of  the  calibration  problems  were  addressed  by  in¬ 
cluding  a  quick  3-step  calibration  process  in  hardware,  but  unfortunately  this  requires  a  bulky 
control  box,  adding  to  the  difficulties  in  using  a  glove  that  is  not  especially  flexible  and  therefore 
a  bit  cumbersome.  The  SuperGlove  has  a  total  of  ten  sensors  and  provides  an  RS-232  interface, 
but  requires  the  control  box  for  the  interface. 


Figure  4;  SuperGlove  from  Nissho  Electronics. 


The  glove  that  is  generally  acknowledged  as  superior  by  researchers  in  the  field  is  the  Cy- 
berGlove®  from  Virtual  Technologies  (http://www.viitex.com/products/hw  products/cvbergiove.htmi')  ■  It  is  available 
as  an  18-sensor  model  or  a  22-sensor  model,  with 

•  two  bend  sensors  on  each  finger  (including  thumb), 

•  four  abduction  (side-to-side  finger  motion)  sensors, 

•  sensors  measuring  thumb  crossover,  palm  arch,  wrist  flexion  and  wrist  abduction, 
and 

•  sensors  to  measure  the  flexion  of  the  distal  joints  on  the  four  fingers  (22-sensor 
model  only). 

The  CyberGlove  is  lightweight  and  flexible,  and  should  be  satisfactory  for  TMR  applications 
from  the  standpoint  of  allowing  the  operator’s  hand(s)  to  still  be  usable  for  handling  a  weapon, 
radio,  etc.  The  standard  interface  is  an  RS-232  serial  connection,  which  is  suitable  for  a  variety 
of  potential  OCU  controllers.  The  COTS  version  requires  an  external  enclosure  for  the  interface 
electronics. 


Figure  5;  CyberGlove  from  Virtual  Technologies. 

A  less  expensive  glove,  the  Fifth  Dimension  Technologies  (http  ://www.5dt.com  )  DataGlove,  is 
available  in  5-sensor  and  14-sensor  configurations,  but  is  bulkier.  It  is  an  instrumented  glove  for 
finger  flexion  and  extension  measurement,  and  it  includes  a  3D0F  sensor  for  hand  orientation 
(see  below).  Earlier  versions  included  a  flexor  strip  for  elbow  or  knee  flexion  measurement, 
which  may  a  potentially  useful  feature.  Like  the  CyberGlove,  the  DataGlove  uses  an  RS-232  se¬ 
rial  interface. 
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For  the  simple  deteetion  of  finger-to-finger  eontaet,  the  FakeSpaee  PINCH™  glove  is  avail¬ 
able  (http://www.fakespacelabs.com/products/pinch.htmO.  The  PINCH  SyStem  USeS  eloth  gloves  with 

eleetrieal  sensors  in  eaeh  fingertip.  It  is  able  to  sense  eontaet  between  any  eombination  of  two  or 
more  digits  by  detecting  a  completed  conductive  path.  As  with  the  other  devices  that  are  more 
commonly  used  in  a  virtual  reality  setting,  no  effort  has  yet  been  made  to  miniaturize  the  inter¬ 
face  electronics,  which  are  housed  in  a  separate  control  box. 


Figure  7:  PINCH  system  from  FakeSpaee. 


One  additional  glove  is  described  below  in  section  2.2.5  in  the  context  of  muscular  motion 
detection,  and  gloves  for  output  feedback  are  described  in  section  2.3.3. 1. 

Most  research  with  gloves  is  related  to  either  virtual  reality  or  gesture  recognition.  It  is  rela¬ 
tively  clear  that  an  “alphabet”  of  gestures  can  be  trained  for  a  given  user  and  that  the  system  will 
perform  adequately  well  for  the  input  of  some  ensemble  of  discrete  commands.  This  is  possible 
using  COTS  software  that  comes  with  the  CyberGlove,  for  example,  and  this  functionality  can  be 
useful  in  a  TMR  OCU  for  menu  navigation,  discrete  command  generation,  etc.  It  is  also  apparent 
that  gloves  are  useful  in  virtual  reality  environments.  Bug  it  is  less  obvious  how  well  they  can  be 
used  for  continuous  control  (e.g.,  teleoperated  driving  and  telemanipulation).  Not  surprisingly, 
the  relevant  research  has  focused  on  telemanipulation,  mimicking  the  motion  of  the  hand  with  a 
robotic  gripper.  At  the  Dextrous  Manipulator  Laboratory  of  Stanford  (http://www- 
cdr.stanford.edu/teiemanip/)  the  CybcrGlovc  has  bccn  uscd  to  control  a  two-fingered  robotic  hand  that 
provides  force  feedback,  under  a  US  Navy  SBIR  program.  In  this  capacity,  it  has  been  a  success 
for  manipulating  objects,  but  the  force  feedback  mechanism  (CyberGrasp™,  discussed  below)  is 
impractical  for  consideration  when  operating  a  TMR  effector-bot. 
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Telemanipulation  of  a  more  dexterous  hand,  the  UTAH/MIT  hand,  was  done  with  a  more 
awkward  device  that  has  evolved  into  the  commercial  Dexterous  HandMaster  (Exos,  Inc.).  Like 
it’s  predecessor,  it  is  essentially  an  exoskeleton  for  the  hand  and  is  not  practical  for  field  use. 

It  is  apparent  that  continuous  control  of  a  vehicle  in  a  driving  mode  with  glove  technology 
(COTS  or  otherwise)  is  an  unknown  entity  and  is  therefore  a  risk  area.  TMR  experimentation 
results  from  Raytheon  are  almost  certainly  the  best  available  information  on  the  use  of  a  glove  in 
controlling  a  mobile  robot.  Later,  there  is  some  discussion  on  the  use  of  gestures  for  controlling 
a  mobile  robot,  which  could  possibly  be  extrapolated  to  consider  gloves  as  the  source  of  the  ges¬ 
ture  information. 

2.2. 1.2  Orientation  Trackers 

Another  commonly-used  device  in  virtual  reality  are  orientation  trackers  that  estimate  the 
roll,  pitch,  and  yaw  of  an  operator  appendage,  usually  the  head  or  hand/wrist.  In  TMR  applica¬ 
tions,  this  probably  has  more  value  in  the  context  the  head  motion,  since  it  is  best  not  to  constrain 
the  hand  orientation  as  a  means  of  input. 

The  CyberTrack  II  (Ligure  8)  is  similar  to  the  the  pitch/roll  sensor  found  on  the  DataGlove 
and  is  also  produced  by  Lifth  Dimension  Technologies,  except  that  it  incorporates  a  magnetic 
compass  for  yaw  output  as  well.  It  is  inexpensive  and  provides  stated  accuracies  of  +!-  2  de¬ 
grees,  although  this  must  surely  be  degraded  for  the  magnetic  compass  under  most  circum¬ 
stances. 


Figure  8:  CyberTrack  II. 


Ascension  (http://www.ascension-tech.com/)  also  manufacturcs  an  orientation  sensor,  the  3D-BIRD, 
shown  in  Ligure  9.  It  has  no  external  electronics  unit  and  plugs  directly  into  an  RS-232  port. 
Like  the  CyberTrack,  it  is  targeted  for  for  head-tracking  applications  in  which  one  needs  to  look 
around  virtual  environments  and  simulated  worlds.  3D-BIRD  makes  up  to  160  measurements  per 
second  with  a  latency  of  about  15  msec,  so  it  is  suitable  for  real-time  tracking.  Angular  accuracy 
is  given  as  2.5  degrees  static  and  4  degrees  dynamic.  Its  dimensions  are  1.34"  x  1.08"  x  1.20" 
(3.4cm  X  2.74cm  x  3.05cm),  and  it  weighs  only  1.2  oz. 
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Figure  9:  3D-BIRD  orientation  tracker. 


2.2. 1.3  Direct-contact  devices 

The  CDG  argues  against  using  any  input  device  that  requires  the  dedicated  use  of  the  opera¬ 
tor’s  hands,  since  they  should  be  free  to  use  a  weapon,  radio,  or  other  device  that  may  be  needed 
to  deal  with  other  situations  as  they  arise.  In  the  strictest  interpretation,  this  would  preclude  the 
use  of  what  we  here  call  direct-contact  devices  (of  the  general  “joystick”  class).  But  for  obvious 
reasons  related  to  ease  of  use,  most  interfaces  for  the  direct  control  of  manipulators  or  mobile 
robots  in  a  visual  servoing  mode  include  some  means  of  physically  directing  the  machine  with 
intuitive  motions.  This  is  usually  implemented  with  devices  such  as  3D  mice,  joysticks,  styli, 
steering  wheels,  and  custom  devices  [Hong  99,  Fong  00,  Kawabata  99].  And  while  visual  ser¬ 
voing  by  a  human  operator  is  an  undesirable  mode  of  operation  from  the  standpoint  of  a  single 
operator  maintaining  control  of  several  robots  simultaneously,  it  must  still  be  accepted  as  a  nec¬ 
essary  evil  in  isolated  (hopefully)  circumstances. 

Consequently,  it  is  incumbent  on  the  OCU  designer  to  consider  if  there  is  some  way  to  take 
advantage  of  this  more  natural  means  of  teleoperating  a  robot  without  impairing  the  operator’s 
ability  to  immediately  have  both  hands  free  as  the  situation  requires.  We  propose  here  that  there 
are,  in  fact,  methods  for  achieving  this.  The  key  elements  of  this  approach  to  using  direct-contact 
devices  in  TMR  applications  include: 

•  Availability  of  small,  low-profile  devices, 

•  Body  (“sewn-on”)  mounting  of  the  device  in  an  unobtrusive,  yet  accessible,  loca¬ 
tion, 

•  Incorporation  of  a  dead-man  switch  on  the  device  to  provide  immediate  context 
switching  to  autonomous  operation  when  the  operator  must  interrupt  teleoperation, 
and 

•  Behavioral  software  support  for  the  dead-man  switch. 

The  last  two  elements  (related  to  the  dead-man  switch)  are  actually  desirable  for  any  means 
of  teleoperating  a  TMR  robot,  not  just  the  proposed  direct-contact  approach.  For  example,  even 
if  it  were  possible  to  teleoperate  a  robot  by  brain  waves  only,  there  must  still  be  some  considera¬ 
tion  given  to  detecting  when  the  operator  is  unable  to  continue  this  mode  of  operation  because  of 
distracting  influences. 
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A  widely  used  device  in  virtual  reality  research  and  CAD  is  the  Spaceball,  currently  pro¬ 
duced  by  Labtec,  which  in  fact  claims  that  it  is  the  “most  widely  used  3D  motion  controller  in  the 
automotive,  aeronautic,  and  consumer  design  industries  worldwide.”  It  allows  intuitive  interac¬ 
tion  with  small  forces  and  limited  motion  of  a  ball  suspended  over  a  base  platorm,  as  shown  in 
Figure  10.  Although  this  particular  implementation  is  larger  than  desired  for  TMR  usage,  there 
are  no  significant  difficulties  in  producing  a  smaller  version.  It  supposedly  is  driftless,  which 
eliminates  the  need  for  periodic  calibration  to  zero  out  biases  that  may  be  interpreted  as  robot 
commands. 


Figure  10:  SpaceBall  3003  FLX  from  Labtec. 


Logitech  manufactures  a  line  of  similar  devices,  including  the  SpaceMouse  XT  (Figure  11a) 
and  CyberPuck  (Figure  11b).  These  devices  (http://www.iogicad3d.com/products/products.htmi)  provide  motion 
control  in  up  to  six  degrees  of  freedom  simultaneously  using  a  disc-shaped  device,  with  a  lower 
profile  than  that  of  the  SpaceBall.  The  sensitivity  is  adjustable,  with  up  to  600  levels.  The  Cy¬ 
berPuck  lacks  the  user-programmable  buttons  of  the  SpaceMouse,  which  could  be  useful  as  part 
of  a  minimalist  user  interface,  but  may  be  difficult  to  package  in  a  wearable  configuration.  No 
calibration  is  necessary  for  these  devices,  and  they  support  either  RS-232  or  USB  connections. 
They  also  require  no  separate  power  input,  drawing  power  instead  from  the  data  connection. 


Figure  1 1:  a)  Magellan/SPACE  MOUSE  "Plus  XT"  (188  x  120  x  44  mm  and  720  grams) 
and  b)  Cyberpuck  (140  x  140  x  45  mm  and  510  grams). 
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A  significantly  different  approach  is  taken  by  Digital  Image  Design,  Inc.  with  their  Cricket 
(http://www.didi.coni/www/areas/products/cricket/').  The  Cricket  (Figure  12)  is  a  3D  input  device,  specifically 
tuned  to  work  in  non-immersive  desktop  Virtual  Reality  environments.  It  provides  6D0F  spatial 
input  along  with  two  conveniently  located  buttons  for  user-defined  functions.  The  buttons  are 
pressure  sensitive,  and  the  thumb  button  (on  the  upper  surface  of  the  handle)  acts  as  a  “flat  joy¬ 
stick,”  sensing  motion  in  three  dimensions.  Perhaps  most  interesting  is  the  incorporation  of  vi¬ 
bration  output  as  a  means  of  providing  tactile  feedback  through  changes  in  amplitude,  frequency, 
and  waveform.  Problems  with  the  Cricket  include  availability  (Digital  Image  Design  appears  to 
only  recently  have  begun  considering  production  of  the  concept),  and  tracking  technology  (it  was 
designed  to  work  in  a  virtual  environment  laboratory  with  external  support).  Nevertheless,  it  is 
interesting  conceptually  and  provides  a  bridge  to  the  next  section,  which  considers  tracking  sys¬ 
tems. 


Figure  12;  Cricket. 


2.2. 1.4  External  observation/tracking  systems 

At  first  glance,  it  would  seem  impractical  to  consider  the  field  deployment  of  systems  that 
have  been  developed  for  the  recognition  of  body  position  and  gestures  in  controlled  environ¬ 
ments  by  observing  the  user  with  external  cameras  or  other  sensors.  But  it  may  be  reasonable  to 
mount  support  equipment  in  the  rear  opening  of  a  HMM’WV  or  other  vehicle.  Such  devices  have 
been  shown  to  be  useful  for  gesture  recognition  in  remote  environments  [Chien  98]. 

The  Polhemus  FASTRAK  system  (http://www.poihemus.com/ftrakds.htm)  is  based  on  the  creation  of 
magnetic  fields  by  a  stationary  transmitter,  which  are  in  turn  sensed  by  a  receiver  mounted  on  the 
user,  with  both  the  receiver  and  transmitter  connected  to  a  control  unit  by  cable.  At  distances  as 
large  as  30  feet,  it  accurately  computes  the  position  and  orientation  the  receiver  as  it  moves 
through  space  in  real  time  with  only  4  msec  latency.  It  can  be  used  with  multiple  receivers,  sacri¬ 
ficing  only  the  position  report  rate.  The  computer  interface  is  high-speed  RS-232  interface  (up  to 
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115.2K  baud)  or  an  optional  IEEE-488  interface  at  up  to  lOOK  bytes/sec.  It  is  not  necessary  to 
maintain  line  of  sight. 


Figure  13:  Polhemus  FASTRAK  system. 


The  Elock  of  Birds  system  from  Ascension  (http://www.ascension-tech.com/products/flockofbirds/flockofbirds.htm) 

is  similar  (Eigure  14),  and  was  originally  developed  for  military  applications.  Each  Elock  receiver 
makes  up  to  144  position  measurements  each  second,  with  latency  of  approximately  15  msec. 
The  DC  field  technology  is  supposedly  more  robust  in  harsh  environments.  A  single  transmit¬ 
ter/controller  can  work  with  up  to  four  units,  as  in  the  case  of  four  robot  operators  each  equipped 
with  tracker  mounted  on  their  head  or  hand,  all  operating  in  the  vicinity  (approximately  10  ft.)  of 
a  single  support  vehicle.  The  host  computer  interface  is  RS-232,  with  selectable  baud  rates  up  to 
115  kBaud.  Static  position  accuracy  is  given  as  0.07"  (1.8mm)  RMS,  and  static  orientation  accu¬ 
racy  is  given  as  0.5°  RMS.  The  receiver  is  less  than  1  cubic  inch. 


Figure  14:  Ascension  Technology  Flock  of  Birds  system. 


Systems  such  as  these  would  likely  be  used  best  to  convey  gestures.  The  use  of  gestures  for 
mobile  robot  operation  has  been  investigated  with  visually-detected  gestures,  which  is  a  more 
difficult  recognition  problem,  but  conceptually  similar.  Eong  et  al.  found  that  such  gestures 
could  be  used  to  teleoperate  a  mobile  robot  (Eigure  15),  but  it  was  not  as  easy  as  would  be  ex¬ 
pected  based  upon  human  experience  with  following  such  gestures  [Eong  00].  They  concluded 
that  it  would  probably  be  necessary  to  add  other  inputs  to  disambiguate  visual  gestures. 
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Figure  15;  The  virtual  joystick  mode  of  the  GestureDriver,  showing  right,  left,  forward  re¬ 
verse,  and  stop  commands  [Fong  00] . 


2.2.2  Keyboards 

Any  keyboard  device  is  a  severe  compromise  to  the  hands-free  requirement  of  TMR.  During 
operation  of  the  OCU  with  robots,  input  of  textual  and  numeric  data  must  be  avoided,  and  this 
should  be  a  key  concept  behind  the  development  of  the  software  that  implements  the  user  inter¬ 
face.  During  breaks  from  active  robotic  control,  it  would  be  advantageous  to  have  the  capability 
of  annotating  logged  data  within  the  OCU,  and  perhaps  performing  other  functions  that  would  be 
much  easier  with  a  full  keyboard. 

One  possibility  would  simply  to  support  the  connection  of  a  standard  PC  keyboard  when  the 
user  is  at  a  command  post  or  some  other  location  where  equipment  is  available.  If  possible,  it 
would  be  preferable  to  have  built-in  capability  for  those  instances  where  the  operator  cannot  re¬ 
turn  to  such  a  facility  between  periods  of  robot  operation.  A  novel  approach  is  a  flexible  key¬ 
board  within  the  operator’s  uniform,  such  as  the  embroidered  approach  shown  in  [PentlandOO] 
(Figure  16).  COTS  possibilities  include  “chord  keyboards”  like  the  Twiddler  (Figure  17),  hand¬ 
held  touch-typing  keyboards  like  the  AlphaGrip  (Figure  18),  and  small  “normal”  keyboards  like 
those  from  Electrone  (Figure  19). 
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Figure  4.  Th«  author  wearing  a  varioty  of  now  dovicas. 

The  glasses  (built  by  MicroopticaJ.  Boston)  contain  a  computer 
display  nearly  invisible  to  others.  The  jacket  has  a  keyboard 
literally  embroidered  into  the  cloth.  The  lapel  has  a  context 
sensor  that  classifies  the  user's  surroundings.  And,  of  course, 
there's  a  computer  (not  visible  in  this  photograph). 
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Figure  16:  Embroidered  keyboard  concept  (and  Microoptical  display),  from  [Pent- 


Figure  17:  The  Twiddler  one-hand  keyboard. 
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Figure  18:  AlphaGrip 


Figure  19:  Electrone  Model  9001  (10.25  inches  wide,  about  60%  of  a  standard  keyboard). 

2.2.3  Speech  Recognition 

Speech  recognition  input  is  undesirable  in  a  TMR  OCU  because  of  background  noise, 
changes  in  voice  characteristics  under  stress,  and  the  general  problems  associated  with  recogni¬ 
tion  technology.  One  such  problem  is  that  there  is  a  tradeoff  between  recognition  reliability  and 
the  degree  of  speaker  independence.  And  without  speaker  independence,  there  is  the  problem  of 
training  the  voice  recognition  software  in  the  field  for  a  specific  user.  This  is  unfortunate,  be¬ 
cause  there  is  evidence  that  voice  control  is  superior  to  manual  control  of  devices  such  as  pan/tilt 
camera  platforms  [Draper  87],  but  speech  recognition  is  still  not  a  hurdle  that  is  worthwhile  for 
TMR  to  take  on. 

Motorola  has  added  a  voice  control  system  as  a  technology  upgrade  to  a  variant  of  the  Land 
Warrior  system,  the  Force  XXI  Land  Warrior  (http://www.mot.com/Gss/ssTG/iSD/ws/vcs.htmi').  In  this  con¬ 
cept,  voice  is  used  for  hands-free  control  of  radios  and  the  display.  Although  targeted  at  high- 
noise  background  environments,  there  is  still  valid  concern  this  sort  of  technology  is  not  yet  suit¬ 
able  for  a  TMR  OCU.  A  “wait-and-see”  stance  is  probably  the  most  appropriate  course  to  take  - 
by  using  some  Land  Warrior  components  (computer  and  radios),  TMR  could  leverage  develop- 
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ment  in  those  areas  and  have  the  optional  use  of  the  voice  control  system,  if  it  proves  to  be  reli¬ 
able,  without  becoming  dependent  on  the  technology. 


W»<yrw<t»erwnhminoorporHd«tw»comK»»iilotop>ion>ln<&lfc»»w«m<litew<«w<i 
ImproM  oommuntoMtom.  Tlw  micioptoM  pioto  up  MiMd  troffl  «*rallom  through  tw  ikul.  M)  ■ 
worts  mo  MglMr  h  iMMutne  an  OKToan  lacwnah  or  noL  LM  bKhpiand  nolM  li  plolitd  up 

>>ymttiilefophono.oothoaghmwtioodr>o<ohou>ot«1s»owohoihooKt 


Figure  20;  Head-contact  microphone  modified  for  firefighter  helmet  use. 


Independently  of  whether  or  not  speech  is  used  for  direct  operator  input,  it  may  be  advanta¬ 
geous  for  the  OCU  to  include  a  complete  audio  system  that  can  be  used  either  with  the  computer 
or  with  wireless  communications  equipment,  multiplexing  the  two  channels  and  providing  the 
best  possible  performance  in  a  noisy  environment.  The  output  speakers  of  such  a  system  will  be 
addressed  later.  A  candidate  input  technology  (htto://www.mtac.pitt.eduAvww/htmi/pg  articie.htmi, 
http://www.stac.ufl.edu/flc/Head%20Microphone.htmi.  dcvcloped  Originally  for  Navy  SEALS  at  the  Coastal  Sys¬ 
tems  Station  of  the  Naval  Surface  Warfare  Center,  utilizes  direct  conduction  of  sound  from  the 
operator  to  the  microphone  through  the  skull,  thus  eliminating  the  need  for  holding  a  microphone 
in  a  particular  place  and  simultaneously  reducing  background  noise  considerably.  This  technol¬ 
ogy  is  being  commercialized  by  Sensory  Devices,  Inc.  for  use  in  firefighter  helmets  (Figure  20). 
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2.2.4  Eye  Trackers 

Discussions  at  Oak  Ridge  National  Laboratory  have  identified  the  importance  of  assessing 
the  operator’s  attention  state.  Eye  trackers  are  one  possible  means  of  achieving  this,  and  they 
also  provide  the  additional  capability  of  acting  as  pointing  devices  for  the  user.  But  the  physical 
size  of  devices  such  as  the  EyeGaze  (EC  Technologies,  Inc.)  and  the  helmet  mounted  iView 
(SensoMotoric  Instruments)  makes  them  impractical  at  this  time,  given  the  limited  benefit  of 
having  them  as  part  of  the  TMR  OCU,  as  shown  in  Eigure  21.  Experimental  platforms  like  the 
visor  used  at  the  University  of  Pennsylvania ‘s  Head-Mounted  Eyetracking  Eab 

(http://www.cis.upenn.edu/-ircs/Trueswellabs/HeadmountedET.htmn  dO  nOt  addrCSS  the  portability  faCtOr,  either. 


Figure  21:  The  portable  version  of  the  EyeGaze  system  (left)  and  the  iView  helmet 
mounted  system  (right)  are  not  suitable  for  a  mobile  warfighter. 


2.2.5  Other 

Ultimately,  an  effective  means  of  controlling  a  robot  may  be  the  use  of  brainwaves,  muscular 
electric  signals  (myography)  or  small,  subtle  movements  of  muscles  such  as  those  in  the  face. 
Efforts  in  this  area  include  those  of  MindTel  and  Sandia  Eaboratories. 

At  MindTel,  under  the  direction  of  David  Warner,  a  series  of  TNG  transducers  (Eigure  22) 
have  been  developed  to  flexibly  interface  a  variety  of  devices  to  computers,  with  a  primary  em¬ 
phasis  on  aids  for  the  disabled.  TNG-1  was  the  first  of  these,  and  was  targeted  at  direct  input  of 
EMG  (electromyographic)  signals  from  facial  or  other  muscles 
(http://www.pui.sar.org/2k/technoiogv/coretech.htmi).  Perhaps  because  of  limited  success  with  EMG  sensing, 
subsequent  TNGs  have  relied  on  more  standard  sensors  (e.g.,  resistive)  that  are  mounted  in  some 
fashion  where  the  user  can  manipulate  them,  even  with  only  limited  muscular  control. 

The  Sandia  research  [Amai  00]  has  attempted  to  actually  control  a  small  mobile  robot  (a 
Mini-RATEERTM)  using  a  commercially-available  device,  the  CyberEink^M  MindMouse.  While 
the  Sandia  team  was  able  to  construct  a  reasonable  mapping  of  detectable  signals  to  robot  control 
functions,  there  are  clear  obstacles  to  using  this  approach  in  the  foreseeable  future.  Brain  waves, 
in  particular,  are  extremely  difficult  to  control  under  stress  or  even  in  the  face  of  minor  distrac- 
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tions.  Facial  muscular  movements  and  the  resulting  signals  are  easier  to  control,  but  difficult  to 
distinguish  enough  for  any  fine  level  of  control.  Without  improved  technology  in  this  area,  robot 
control  is  likely  to  consist  primarily  of  overt  (almost  discrete)  commands  generated  by  general 
muscular  motion  within  a  fairly  large  region  of  the  face  (e.g.,  movement  almost  anywhere  in  the 
jaw/neck  region  could  be  interpreted  as  a  single  discrete  command).  This  corresponds  in  many 
respects  with  commanding  a  robot  with  cursor  keys,  and  as  noted  earlier,  discrete  “cursor-key” 
approaches  to  robot  control  are  not  particularly  promising. 


Figure  22:  TNG-3B,  the  most  recent  sensor  interface  from  MindTel. 


[Robinson  98]  also  describes  some  of  the  limitations  of  electromyography  as  a  means  of  ro¬ 
botic  teleoperation.  Although  a  potential  site  on  the  lower  arm  is  identified  as  to  provide  a  range 
of  specified  control  signals,  they  also  recognize  practical  difficulties  with  EMG  control. 

Given  that  a  TMR  OCU  may  want  to  input  from  various  muscles  and  would  probably  take 
the  more  reliable  course  of  detecting  movement  directly  rather  than  by  means  of  EMG,  it  may  be 
useful  to  consider  resistive  touch  sensors  such  as  those  from  Infusion  Systems 
(http://www.Infusionsvstems.co  m/).  These  devices  are  targeted  at  musicians  for  the  control  of  instruments, 
lighting,  and  effects  during  performance.  The  basic  Touch  Sensor  is  a  1.7”xl.7”  touch  pad,  with 
a  1-2  msec  response  time  and  capable  of  sensing  pressure  over  a  range  of  100  grams  to  10  kg. 
With  identical  specifications,  mini-  and  micro-  versions  are  available  as  0.5”  and  0.2”  diameter 
circles,  respectively.  A  24”  x  0.5”  strip,  trimmable  to  user  requirements,  is  also  available. 

Six  of  these  touch  sensors  are  included  in  the  Infusion  Systems  TouchGlove,  designed  as  a 
controller  for  drum  machines.  One  sensor  is  mounted  in  each  fingertip,  and  one  is  mounted  in 
the  palm.  Since  the  TMR  operators  hands  are  already  in  demand,  it  may  be  more  appropriate  to 
consider  the  use  touch/pressure  sensors  elsewhere,  such  as  in  the  shoes  where  they  can  be  acti¬ 
vated  by  toe  movements. 

Einally,  we  note  that  relative  to  operator  attention  and  perhaps  to  operator  stress  level,  it  may 
be  useful  to  assess  certain  physiological  conditions,  such  as  heart  rate,  breathing  rate,  and  gal- 
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vanic  skin  response.  This  is  a  lower  priority  with  respect  to  initial  deployment  and  will  not  be 
considered  in  detail  at  this  time.  Emerging  technologies  such  as  the  “Smart  Shirt” 
(http://vishwa.tfe.gatech.edu/gtwm/gtwm.htmil  providc  possibilities  for  nonobtrusivc  monitoring  of  vital  signs, 
utilizing  optical  and  conductive  fibers  woven  inside  the  garment. 

2.3  Output  Device  Technolosies 

There  is  tendency  to  rely  primarily  upon  visual  output  to  convey  information  to  any  remote 
operator  of  autonomous  or  teleoperated  systems,  mainly  because  of  the  high  information  band¬ 
width  of  visual  displays.  The  importance  of  non-visual  feedback  is  stated  in  the  context  of 
UAVs  in  AFRL  research  [Draper  00],  in  which  it  is  noted  that  detecting  turbulence  visually  is 
probably  too  slow  or  too  unreliable  for  effective  and  safe  control.  This  is  not  to  say  that  non¬ 
visual  feedback  cannot  be  conveyed  by  visual  means,  and  the  AFRL  recommendations  in  fact 
include  the  display  of  other  data  (e.g.,  orientation  and  attitude)  by  HUD  techniques.  The  follow¬ 
ing  sections  first  consider  the  available  technologies  for  non-distracting  visual  displays  and  then 
other  means  of  providing  operator  feedback. 

2.3.1  Displays 

Prior  efforts  have  shown  that  real-time  video  is  the  single  mechanism  most  relied  upon  for 
feedback  when  it  is  necessary  to  take  control  of  a  robot,  either  one  designed  for  dedicated  teleop¬ 
eration  [Draper  95,  Hainsworth  00,  Wettergreen  97,  Ballou  00,  Laird  00,  Power  00]  or  for 
teleautonomous  (telereflexive)  operation  [Bares  97,  Fong  00,  Gilbreath  00].  For  this  application 
alone,  if  nothing  else,  it  is  necessary  to  survey  and  evaluate  displays  that  may  be  feasible  for  field 
operation  in  TMR  scenarios.  Based  on  this,  and  in  conjunction  with  other  TMR  requirements,  we 
can  formulate  three  principles  that  can  guide  the  survey  of  video  display  technology: 

1.  It  is  necessary,  at  least  in  isolated  instances,  to  display  full  video  frames  (camera 
views,  maps,  etc.), 

2.  It  is  desirable,  MOST  of  the  time,  to  display  augmented  reality  data  in  the  least  obtru¬ 
sive  manner  possible  (still  allowing  the  operator  to  “see  through”  the  data),  and 

3.  At  some  intermediate  level  of  frequency,  it  MAY  be  necessary  to  display  GUI  inter¬ 
faces,  such  as  menus  and  dialog  boxes. 

The  need  to  satisfy  these  constraints  with  a  heads-up  display,  minimizing  the  inconvenience 
of  managing  two  separate  visual  inputs  at  once,  has  been  noted  by  various  researchers,  including 
[Stamer  95]. 

One  example  of  an  impractical  and  distracting  display  is  the  early  Land  Warrior  prototype 
shown  in  Figure  23.  Although  it  can  be  swung  out  of  the  way  (which  may  be  suitable  for  inter¬ 
mittent  use  by  soldiers  in  general),  this  is  not  desirable  for  a  soldier  interacting  almost  constantly 
with  a  group  of  robots  while  still  maintaining  awareness  of  his  surroundings.  And  while  one 
could  argue  that  such  a  display  is  transparent  in  the  sense  that  one  eye  still  can  see  beyond  it,  ex¬ 
perience  indicates  that  this  is  difficult  in  practice. 
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Figure  23:  Early  Land  Warrior  helmet  mounted  display  that  clearly  exhibits  some  of  the 
features  to  be  avoided. 

Newer  developments  in  wearable  display  technology  are  less  cumbersome  and  provide  some 
degree  of  transparency  (and  admittedly  are  less  ruggedized).  But  even  casual  observers  can  see 
the  problem.  At  a  gathering  of  wearable  computer  enthusiasts  and  vendors,  CNN  reporter  Ann 
Kellan  observes  “One  eye  on  the  monitor,  one  eye  on  the  real  world,  may  be  natural  for  these  te¬ 
chies,  but  it  takes  getting  used  to,”  noting  that  it  takes  a  conscious  effort  to  pay  attention  to  real¬ 
ity. 

2. 3. 1.1  Head  Mounted  Displays 

Head  mounted  displays  have  been  used  in  military  applications  for  many  years  already,  but 
suffer  from  criticisms  of  awkwardness,  visual  discomfort,  fragility,  and  vision  obstruction.  As 
long  ago  as  1995,  however,  Starner  et  al.  [Starner  95]  argued  that  such  displays  are  victims  of 
several  misconceptions.  First,  they  note  that  focus  depth  is  not  a  problem,  since  many  units  have 
adjustable  focus  depth,  and  this  can  be  set  to  provide  a  level  of  comfort  exceeding  typical  fixed 
display  monitors.  There  is  an  implicit  assumption  here,  arguably  not  valid  for  TMR,  that  the  user 
will  have  a  somewhat  constant  focus  depth  in  the  real  world.  Supposedly  this  assumption  holds 
true  for  many  everyday  situations,  such  as  walking  down  the  street,  sitting  in  a  meeting,  etc. 

Second,  Starner  et  al.  note  that  these  displays  do  not  act  as  an  “eye  patch,”  with  the  display 
image  possibly  overriding  the  real  world,  and  instead  allow  most  users  to  easily  merge  the  im¬ 
ages  from  both  eyes.  They  do  acknowledge,  however,  that  the  greater  the  degree  of  difference 
between  display  content  and  real-world  content,  the  more  difficult  this  may  be.  And  finally,  they 
argue  that  there  is  no  significant  adaptation  period  when  taking  these  displays  on  and  off. 

Improvements  in  head  mounted  displays  (HMDs)  have  resulted  in  those  such  as  the  Land 
Warrior  HMD,  developed  by  Kaiser  Electronics,  as  shown  in  Figure  24.  Devices  such  as  this  are 
rugged  enough  for  military  use,  and  they  attempt  to  avoid  the  distraction  issue  by  allowing  them 
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to  be  moved  aside  completely  or  at  least  out  of  the  direct  line  of  sight,  as  in  Figure  24(b).  Com¬ 
prised  of  an  800  x  600  pixel  resolution  LCD,  the  unit  provides  a  40-degree  FOV. 

Initial  experiences  with  these  displays  in  warfighting  experiments  are  encouraging.  Also,  the 
use  of  an  HMD  in  the  MPRS  (Man  Portable  Robotic  System)  seemed  to  cause  no  difficulties  dur¬ 
ing  exercises  in  which  robots  were  teleoperated  in  tunnels,  aside  from  the  reported  desire  of  sec¬ 
ondary  operators  to  also  see  the  video  and  some  difficulties  in  bright  sunlight  [Laird  00,  Bruch 
00]. 


Figure  24:  Land  Warrior  HMD  developed  by  Kaiser  Electronics,  (a)  side  profile  (b)  in  use 

while  aiming  a  weapon. 


Alternatives  to  the  rugged  Land  Warrior  HMD  exist  providing  other  attractive  characteris¬ 
tics.  TekGear  (www.tekgear.ca)  is  producing  the  M2,  a  high  end  portable  viewing  device  targeted  at 
mission  critical  display.  The  integral  LCD  provides  800  x  600  resolution  in  full  color.  The  base 
model  shown  in  Figure  25  is  intended  to  be  a  universal  form  factor  and  can  be  modified  to  OEM 
configurations  (e.g.,  helmet-mounted). 


Figure  25:  TekGear  M2,  a)  complete  headset  version,  and  b)  details  of  LCD  display. 
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TekGear  M2  Specifications 

Field  of  View 

22°  Horizontal,  16.6°  Vertical 

Optics 

High  Efficiency,  See  Through 

Video  Resolution 

480,000  pixels  (800  x  600) 

Pixel  Pitch 

12  Microns 

Input  Siqnal 

24  bit  Digital  RGB  GVIF  interface 

Video  Refresh  Rate 

60Hz 

Contrast 

50:1 

Operating  Temperature 

0  to  50  degC 

Storage  Temperature 

-10  to  +80  degC 

Shock  and  Vibration 

Industrial  Environment 

Total  Weight 

21  Og 

Controls 

Brightness,  Image  Flip 

Power  Consumption 

Under  2.0W 

Input  Voltage 

5V  DC 

MicroOptical  Corporation  (http://www.microopticaicorp.com/')  has  developed  an  “Invisible  Monitor” 
technology  that  can  be  clipped  on  as  in  Figure  26(a)  or  integrated  into  eyeglasses  as  in  Figure 
26(b).  The  C-1  clip-on  field  test  kit  contains  one  Invisible  Monitor  Clip-on  information  display 
with  “see-around”  display  optics,  articulating  mounting  arm,  and  'VGA  and  NTSC  conversion 
electronics.  The  housing  of  the  conversion  electronics  is  separated  from  the  display  by  a  4-foot 
cable.  The  Integrated  EyeGlasses  contain  simlar  optics,  cabling,  and  conversion  electronics. 


Figure  26;  MicroOptical  C-1  Clip-On  display  (a)  and  integrated  eyeglasses  (b). 
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Figure  27:  Thad  Stamer  from  Georgia  Tech  wearing  the  MicroOptical  glasses. 


MicroOptical  Specifications 

Display  Format 

320x240,  16-Bit  Color 
(640x480  under  development) 

Refresh  Rate 

60  Hz 

Field  of  View 

Approximately  1 1  degrees  horizontal 

Optics 

Left  eye  version  is  available  now,  right  eye  versions 
by  special  order 

Focus  Range 

Pre-set  to  1  meter  (other  distances  available  upon 
request) 

Eyeglass  Frame  Size 

Clip-on  fits  most  wire  frame  glasses.  Plano  glasses 
supplied 

Video  Input 

Standard  VGA,  female  DB-15  connector  and  stan¬ 
dard  NTSC,  RCA  plug 

Power  Requirements 

9  V,  100  mW  (display  and  backlight). 

9  V,  2.9  W  (VGA  interface). 

9  V,  1 .9  W  (NTSC  interface) 

4.5  V,  300mW  (RSI  70). 

Operating  Temperature 

32  degrees  to  1 04  degrees  F  (0  degrees  to  40  de¬ 
grees  C) 

Storage  Temperature 

-4  degrees  to  104  degrees  F  (-20  degrees  to  40 
degrees  C) 

Head-Supported  Weight 

Clip-on:  33  grams 

Integrated  EyeGlasses:  62  grams 

2.3. 1.2  Retinal  Displays 

Even  the  best  HMD  implementations  have  several  problems,  including  limited  field  of  view 
(without  sacrificing  real  field  of  view),  size,  weight,  resolution,  brightness,  lack  of  true  transpar¬ 
ency,  and  power  consumption.  A  promising  approach  to  solving  these  problems  of  traditional 
HMDs  is  the  virtual  retinal  display  (VRD).  Invented  at  the  HIT  lab  of  the  University  of  Wash¬ 
ington  (http://www.hiti.washinptnn.edu/research/vrd/i  ill  1991,  VRDs  do  uot  rcquirc  the  sizc  aud  cucrgy  associ¬ 
ated  with  actually  forming  an  image  on  a  screen,  instead  directly  scanning  the  image  onto  the  ret¬ 
ina  in  the  most  efficient  way  possible.  The  technology  is  being  commercialized  by  Microvision 
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(http://www.  mvis.com/')  as  the  Retinal  Scanning  Display  (RSD).  In  keeping  with  the  intent  herein  of 
providing  an  interface  more  akin  to  augmented  reality,  the  RSD  is  targeted  at  applications  in 
which  it  is  necessary  to  display  an  image  that  is  overlaid  and  does  not  block  a  user’s  view.  Ini¬ 
tially,  this  is  monochrome,  but  with  technology  improvements  it  will  be  possible  to  provide  full- 
color  displays.  According  to  a  recent  trade  publication  article,  “adding  green  and  blue  is  only  a 
matter  of  time,  especially  since  funding  from  the  commercial  and  military  industries  is  ample, 
and  the  Army,  Navy  and  Air  Force  are  all  investigating  the  RSD  for  use  in  next-generation  AR 
HMD  systems”  [Ciufo  00] . 


Figure  28:  Viewing  a  planar  image  (a)  vs.  the  Retinal  Scanning  Display  approach  (b). 

RSD  relies  upon  low-intensity  lasers  and  microelectromechanical  (MEM)  devices  to  scan  an 
image  directly  onto  the  retina  of  the  eye,  as  shown  in  Figure  28.  This  completely  eliminates  the 
need  for  an  image  display  such  as  an  LCD,  as  well  as  the  optics  required  to  allow  the  user  to  fo¬ 
cus  on  the  close  device.  Since  the  image  characteristics  can  be  controlled  directly  by  a  host  com¬ 
puter,  it  is  possible  to  vary  the  size  or  apparent  distance  as  needed.  Although  initial  versions,  like 
that  shown  in  Figure  29,  rival  other  HMDs  in  physical  size,  it  should  be  possible  to  immediately 
reap  the  benefits  of  a  sharp,  clear,  transparent  display. 


Figure  29:  Nomad  prototype  RSD  headgear  from  Microvision. 
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Retinal  Scanning  Display  Specifications 

Resolution 

VGA  (640  X  400  pixels) 

SVGA  (800  X  600  pixels*) 

Luminance 

1  -480  FL  at  the  eye  (500h-  FL*) 

Grayscale 

32  user  discernable  gray  shades 

Dimming  Ratio 

1000:1 

Field  of  View: 

30  degrees  horizontal,  22  degrees  vertical,  ap¬ 
proximately  equivalent  to  a  19"  Monitor 

Display  Color: 

Monochrome  red  (635  nm) 

Refresh  rate 

60  Hertz 

Interface 

SVGA*,  VGA,  NTSC*,  PAL* 

HMD  weight 

<  1 .5  lb.  (657g) 

Power 

12  VDC 

*  in  development,  available  mid  2001 

For  all  of  its  promise,  RSD  technology  is  not  without  its  risks.  In  addition  to  the  fact  that 
commercial  deliveries  have  not  begun,  and  some  of  the  specifications  pertain  to  upcoming  im¬ 
provements,  other  possible  problems  include  difficulty  in  maintaining  proper  alignment  of  the 
exit  pupil  with  the  eye,  cost  reduction,  and  ruggedization. 

2.3. 1.3  Omnidirectional  camera  imagery 

TMR  has  accepted  the  value  of  omnidirectional  (actually,  panospheric  is  probably  a  better 
description)  camera  imagery  for  typical  missions.  Development  of  the  OmniCam  at  Columbia 
University  (http://www.cs.coiumbia.edii/CAVF./nninicam/l  rcsultcd  in  the  licensing  of  the  technology  to  Re- 
moteReality  (http://www.remotereaiitv.com) ■  formerly  CycloVision  Technologies.  Although  other  omnidi¬ 
rectional  approaches  had  been  implemented,  this  was  the  first  to  provide  a  single  effective  view¬ 
point  [Baker98].  Such  a  viewpoint  preserves  perspective  in  all  directions,  and  thus  makes  it  pos¬ 
sible  to  choose  a  direction  to  look  at  any  time  and  see  it  from  exactly  the  same  point  in  space.  In 
robotic  applications,  prior  work  with  NASA’s  Nomad  [Wettergreen  97]  and  the  Ames  Marsok- 
hod  Rover  [Christian  97]  had  established  the  usefulness  of  this  data  for  a  variety  of  applications 
including  teleoperation,  exploration,  reconnaissance,  or  searching. 

From  the  standpoint  of  OCU  design,  the  important  issue  is  the  presentation  and  use  of  omni¬ 
directional  imagery.  Ongoing  research  at  the  University  of  Pennsylvania  has  addressed  this 
within  the  context  of  TMR  and  similar  applications  (http://www.cis.upenn.edu/-kostas/omnigrasp.htmi).  Some 
of  the  most  relevant  work  on  utilization  of  omnidirectional  images  for  teleoperation  has  been 
done  at  Lehigh  University  (http://www.eecs.iehigh.edu/-tbouit/remote-reaiitv.htmi).  Vehiclcs,  including  the  one 
shown  in  Figure  30,  have  been  teleoperated  by  presenting  users  with  interfaces  combining  multi¬ 
ple  unwarped  images  [Power  00].  Usability  studies  have  indicated  some  disorientation  problems, 
and  not  surprisingly  the  forward  view  is  most  suitable  for  normal  driving  [Boult  00] . 


28 


Figure  30:  A  vehicle  built  at  Lehigh  for  teleoperation  with  omnidirectional  imagery. 


2.3.2  Audio 

The  operator  must  remain  in  voiee  eontaet  with  other  military  personnel,  and  it  is  straight¬ 
forward  to  allow  any  OCU  audio  outputs  to  reaeh  the  operator  through  a  shared  output  device. 
Intuitively,  it  illogical  to  use  an  earphone  or  headphone,  since  this  would  mask  local  sounds  that 
may  be  important  cues  of  impending  danger  to  the  operator.  Consequently,  direct  auditory  con¬ 
duction  through  facial  bones  appears  to  be  a  more  appealing  prospect.  But  although  studies  have 
shown  that  bone-conduction  speech  recognition  thresholds  are  similar  to  normal  air-channel 
thresholds  [Edgerton  77],  and  even  though  noise-masking  characteristics  are  also  similar  in  the 
two  mechanisms  [Robinson  82],  there  are  few  COTS  products  that  directly  use  conduction,  aside 
from  hearing  aids.  According  to  the  Military  Audiology  Association,  research  continues  in  this 

area  (http://www.miiitarvaudioiogv.org/newsietter02/aac.htmO.  but  little  published  data  is  available. 


2.3.3  Tactile  and  Haptic  devices 

One  of  the  most  widely  adopted  technologies  for  tactile/force  feedback  is  the  TouchSense 
technology  available  from  Immersion  (http  ://www.immersion.coTn  ).  This  has  been  incorporated  into  a 
variety  of  entertainment  products  (mostly  joysticks,  steering  wheels,  or  variants  thereof)  and  sev¬ 
eral  medical  products.  Medical  applications  include  training,  such  as  the  simulation  of  the  sensa¬ 
tion  of  insertion  of  a  needle,  catheter,  endoscope,  or  other  medical  device  into  a  human  subject 
(HT  Medical).  Analogous  sensations  for  a  robot  operator  could  include  the  “skin  resistance”  as 
the  robot  attempts  to  penetrate  a  door  or  other  opening,  the  “pierce- and-relax”  sensation  as  the 
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robot  succeeds,  and  the  “steady  drag”  as  the  robot  moves  down  a  hall  or  other  close-quarters 
situation. 


SPAWAR  researchers  have  uses  a  vibrating  pager- like  device  inside  a  pendant  controller  to 
directly  provide  velocity  feedback  [Gilbreath  00].  Hong,  et  al.  demonstrated  force  feedback  to 
teleoperate  a  mobile  robot  with  a  joystick  over  both  level  ground  and  stairs  [Hong  1999].  The 
reflected  force  corresponded  to  the  potential  field  of  nearby  obstacles  in  the  level  ground  case 
and  to  the  impulse  force  of  climbing  when  on  stairs. 

Fong  et  al.  [Fong  00]  have  used  the  Delta  Haptic  Device  (Figure  31),device  developed  at  the 
Ecole  Polytechnique  Federale  de  Lausanne  (Swiss  Federal  Institute  of  Technology)  to  provide 
feedback  during  teleoperation  of  a  small  Koala  Robot  (Figure  32).  The  robot  is  equipped  with 
short-range  proximity  sensors,  and  the  HapticDriver  interface  transmits  to  the  operator  a  force 
proportional  to  the  closeness  of  detected  obstacles.  Trials  with  test  subjects  (complete  novices  at 
a  trade  show)  showed  that  the  force  feedback  often  made  the  difference  in  being  able  to  drive 
through  a  maze  without  collisions.  It  was  determined  to  be  an  “effective  interface  for  navigating 
cluttered  environments  and  for  performing  docking  maneuvers.”  Unfortunately,  the  Delta  Haptic 
Device  is  impractical  for  field  deployment,  and  it  is  not  obvious  how  a  portable  device  could  be 
constructed  to  convey  that  degree  of  force  information. 


Figure  31;  The  Delta  Haptic  Device. 
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The  HapticDriver  currently  uses  only  2D  force  information,  but  the  developers  are  consider¬ 
ing  the  possibility  of  mapping  wheel  torques,  3D  orientation,  or  accelerations  into  3D  forces, 
perhaps  allowing  operator  perception  of  driving  across  uneven  terrain  or  through  minor  obstruc¬ 
tions,  humps,  etc. 


Figure  32:  Koala  robot  used  for  teleoperation  with  haptic  interface. 


2.3.3. 1  Gloves  for  Haptic  and  Tactile  Feedback 

True  haptic  feedback  is  not  an  option  for  a  man-portable  configuration  at  this  time.  Devices 
like  the  Delta  Haptic  Device  described  above  are  far  too  cumbersome.  The  least  objectionable 
implementation  is  the  CyberGrasp,  from  Virtual  Technologies,  the  maker  of  the  CyberGlove  de¬ 
scribed  earlier  (http  ://www.  virtex.com).  Designed  to  be  used  in  conjunction  with  the  22-sensor  version 
of  the  CyberGlove,  the  CyberGrasp  is  a  massive  cable  driven  mechanism,  clearly  not  practical  for 
a  TMR  OCU.  Furthermore,  it  requires  a  6D0F  sensor  like  the  Flock  of  Birds  or  Polhemus  sys¬ 
tems  discussed  earlier. 

Of  greater  interest,  and  also  manufactured  by  Virtual  Technologies,  is  the  CyberTouch  op¬ 
tion  for  the  CyberGlove.  This  introduces  small  vibrotactile  stimulators  into  each  finger  and  on 
the  palm  of  the  glove.  The  stimulators  are  individually  addressable  and  support  both  pulsed  and 
sustained  vibration.  According  to  Virtual  Technologies,  it  is  possible  to  achieve  the  perception 
of  touching  a  solid  object,  which  could  of  course  be  useful  if  using  a  glove  to  teleoperate  a  robot. 
Even  aside  from  such  a  capability,  the  vibrotactile  stimulators  provide  a  means  of  signaling  the 
operator  without  requiring  visual  attention.  This  could  be  useful  both  during  continuous  control 
of  a  robot  (to  convey  remote  sensory  information  or  robot  status)  or  simply  to  get  the  operator’s 
attention  as  an  “interrupt.”  Specifications  of  the  CyberGlove  include  a  vibrational  frequency 
range  of  0-125  Hz,  a  vibrational  amplitude  of  1  N  peak-to-peak  @125  Hz,  and  a  weight  of  5  oz. 
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2.4  Support  Technolosies 

2.4.1  Cabling  and  Communication 

The  larger  issues  of  reliable  eommunieation  between  robots,  warfighters,  and  eommand 
posts  will  not  be  addressed  here,  and  in  fact  are  the  subject  of  other  programs  with  much  wider 
scope  than  robotics.  For  the  purposes  of  near-term  demonstrations,  the  working  assumption  must 
be  that  wireless  communication  between  agents  (robotic  or  human)  will  be  much  as  it  has  been  in 
the  2000  TMR  demonstration,  a  mixture  of  wireless  point-to-point  modems  and  standard  WLAN 
(wireless  local  area  network)  technology  with  add-on  equipment  to  improve  performance.  To  ad¬ 
dress  early  deployment  of  robotic  units  and  the  need  for  improved  LPFLPD  performance,  this 
will  have  to  be  replaced  with  emerging  military  radio  technology,  possibly  from  the  Land  War¬ 
rior  program  or  a  new  “lightweight  warrior”  ATD  (advanced  technology  demonstration)  that 
could  begin  in  2000-2001  [Erwin  00]. 

Of  greater  interest  for  the  design  of  a  near-term  OCU  is  the  interconnection  of  the  devices 
that  are  proposed  for  an  individual  warfighter.  Most  of  the  devices  can  be  interconnected  with 
cables  that  pass  through  sewn  loops  on  the  uniform,  minimizing  the  possibility  of  snagging. 
Wireless  Bluetooth  technology  for  peripherals,  where  available,  is  certainly  a  possibility  to 
minimize  the  use  of  cables.  Because  of  its  short  range,  it  is  less  of  a  risk  for  a  deployed  system, 
but  should  still  be  used  only  with  caution  and  after  prior  testing.  Infrared  peripherals,  using  pro¬ 
tocols  such  as  IRDA,  are  generally  not  desirable  because  of  their  requirement  that  line  of  sight  be 
strictly  maintained. 

But  since  it  is  necessary  to  provide  cables  to  supply  power  to  wireless  peripherals,  it  is  argu¬ 
able  whether  any  significant  advantage  is  gained  by  not  using  the  same  cable  route  to  convey  sig¬ 
nal  information.  This  is  especially  true  of  some  of  the  RS-232  serial  devices  which  can  be  self- 
powered  off  of  the  wired  data  interface. 

2.4.2  CPUs 

As  noted  earlier,  wearable  computing  technology  is  the  logical  source  to  ultimately  provide 
COTS  components  for  the  heart  of  the  OCU.  In  the  short  term,  for  the  sake  of  ruggedness,  there 
are  distinct  advantages  in  considering  the  Computer/Radio  Subsystem  (C/RS)  of  the  Land  War¬ 
rior  program,  developed  by  Motorola  (http://www.mot.com/Gss/ssTG/iSD/ws/wamorcrs.htmi).  Although  more 
bulky  than  the  sleekest  units  currently  available  from  vendors  like  Xybemaut  and  VIA,  the  C/RS 
is  still  only  3.2  pounds  (with  radio,  frame  grabber,  and  GPS)  and  provides  a  variety  of  interfaces, 
including 

•  VGA  or  RS-170  for  Helmet  or  Hand  Held  Display  with  Touch  Screen 

•  Thermal  Weapons  Sight  or  Video  Camera  video  input 

•  Radio  Interfaces,  2  internal,  1  external 

•  4  Channel  Laser  Detector 

•  PCI  and  ISA  Bus  for  future  expansion 

•  Keyboard/mouse 

•  2  ea.  RS-232 
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•  Ethernet 

•  USB 

Physical  size  is  10.6  x  7.0  x  1.7  in.  for  the  computer  and  4.5  x  3.0  x  1.5  in  for  the  radio,  and 
is  designed  to  be  carried  within  a  backpack  frame. 

2.5  Operator  Interface  Design  and  Integrated  Augmented  Reality 

The  importance  of  an  augmented  reality  (AR)  model  to  describe  the  conveyance  of  remote 
information  in  a  TMR  OCU  has  already  been  discussed.  In  this  section,  we  elaborate  on  some 
notional  concepts  for  the  actual  implementation  of  the  operator  interface  and  the  incorporation  of 
AR  techniques.  This  is  necessary  to  develop  a  complete  concept  of  a  recommended  hardware 
implementation  in  the  next  section. 

The  nature  of  the  robot  status  information  being  displayed  in  prior  efforts  to  develop  OCUs 
tends  to  depend  on  the  criticality  of  the  robot  (its  intrinsic  value,  relative  to  the  risk  in  which  it  is 
placed)  in  conjunction  with  the  “cost”  of  displaying  such  information.  In  cases  where  robots  are 
sent  into  dangerous  situations,  such  as  advance  exploration  of  mines  [Hainsworth  00]  or  space 
applications  [Lane  00a,  Wettergreen  97,  Christian  97,  Nguyen  00]  and  where  multiple  large  for¬ 
mat  displays  are  available,  informational  displays  become  very  extensive.  Since  such  situations 
normally  fit  the  “Mission  Control  Station”  model  of  an  OCU,  often  with  multiple  operators 
available  to  monitor  mission  status,  it  is  quite  understandable  to  build  displays  that  provide  all 
information  that  may  possibly  be  useful. 

NASA  Ames  has  developed  the  Virtual  Environment  Vehicle  Interface 
(http://img.arc.nasa.  gov/vEvi/index.html)  for  the  purposc  of  rcmotcly  teleoperating  robotic  vehicles  such  as 
Nomad  [Wettergreen  97]  and  the  Ames  Marsokhod  Rover  [Christian  97].  This  approach,  how¬ 
ever,  takes  the  extreme  position  of  immersing  the  operator  in  a  complex  representation  of  the  ro¬ 
bot’s  environment  in  order  to  maximize  the  probability  of  successful  operation,  since  the  robot  is 
extremely  valuable,  latency  is  high,  and  the  environment  is  typically  not  well  known.  But  given 
the  available  bandwidth  for  display  of  data,  visual  or  otherwise,  sacrifices  must  inevitably  be 
made  in  the  status  information  provided  by  a  TMR  OCU.  It  cannot  be  “designed  by  committee” 
to  accommodate  the  wishes  of  all  designers  and  potential  users,  as  difficult  as  the  decisions  may 
be  to  defend  later  with  the  benefit  of  hindsight. 

Even  if  the  adverse  effects  (primarily  distraction)  of  a  teleimmersive  environment  could  be 
tolerated,  it  is  interesting  to  note  that  some  research  suggests  that  it  may  not  even  be  that  effec¬ 
tive  of  an  approach  for  many  tasks.  In  applications  involving  remote  driving  and  operation  of 
heavy  equipment,  researchers  at  the  Helsinki  University  of  Technology  [Halme  00]  discovered 
that  the  use  of  a  head  tracker  with  a  pointable  camera  was  sometimes  helpful  in  choosing  direc¬ 
tion  in  unfamiliar  terrain,  but  not  as  useful  when  moving  in  a  specific  direction,  as  when  being 
constrained  by  walls  or  tunnels.  The  use  of  a  HMD  in  any  of  these  applications  was  found  to  be 
stressful  and  could  cause  nausea. 

In  the  context  of  the  MPRS  program,  SPAWAR  researchers  also  note  the  taxing  nature  of 
teleoperation  through  onboard  video,  often  accentuated  by  problems  with  contrast  and  lighting 
[Laird  00].  The  results  of  MPRS  tests  with  military  users  at  Port  Leonard  Wood  are  somewhat 
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limited  in  their  applicability  to  the  primarily  autonomous  nature  of  TMR,  since  most  users  were 
uncomfortable  with  any  degree  of  autonomy  in  the  MPRS  systems. 

After  experiments  with  Dante  II  [Bares  97],  the  following  guidelines  emerged  with  respect 
to  the  interface  design: 

“Consistent  appearance  and  interaction:  Similar  or  identical  design  throughout 
the  interface  allows  operators  to  focus  on  robot  actions  rather  than  the  mechanics 
of  using  the  interface. 

“Functional  organization:  It  is  appropriate  to  embed  the  functional  layout  within 
the  interface  to  avoid  operator  confusion.  The  use  of  operational  control  contexts 
pro-vides  a  unifying  and  simplifying  perspective  on  human-machine  interaction. 

This  approach  enabled  us  to  concisely  organize  the  interface  so  that  commands 
appropriate  for  a  particular  type  of  function  are  grouped  together. 

“Uncluttered  layout:  Clean  graphical  design  with  qualitative  or  simple  quantita¬ 
tive  representations  of  sensor  and  state  information  allows  quick  assessment  of 
current  conditions.  Numeric  data  provides  precision  and  should  support  graphical 
features  unobtrusively. 

“Simple  command  generation:  Clear,  easy-to-use  controls  allow  efficient,  rapid 
command  sequences.  Easily  modified  values  and  reusable  commands  are  impor¬ 
tant  for  reducing  operator  workload  during  teleoperation. 

“Visual  indication  of  safeguards:  Different  command  safeguards  are  appropriate 
depending  upon  the  situation  and  the  types  of  commands  being  applied.  Indicators 
that  clearly  reflect  active  safeguards  reduce  operator  misconceptions  and  error.” 

[Bares  97] 

Mobile  robot  operator  interfaces,  whether  for  teleoperation  or  supervisory  control,  and  re¬ 
gardless  of  the  operational  domain  (air,  ground,  underwater,  space)  historically  have  several  ma¬ 
jor  modes.  While  there  are  differences  in  nomenclature  and  in  implementation,  it  is  only  a  minor 
oversimplification  to  say  that  most  prior  work  advocates  the  use  of  at  least  four  primary  modes: 

•  Sensor/status  mode, 

•  Command  mode, 

•  Robot  perspective  mode,  and 

•  Map  mode. 
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Figure  33:  Multimodal  display  of  the  PdaDriver,  showing  their  “video,”  “map,”  “com¬ 
mand,”  and  “sensor”  modes  [Fong  00]. 


Archetypal  examples  of  multimodal  interfaces  for  teleoperation  of  mobile  robots  include  the 
Virtual  Dashboard  of  the  NASA  Nomad  [Wettergreen  97],  the  PdaDriver  [Fong  00],  and  TMR 
concepts  [Bay  00].  Depending  on  display  constraints,  some  were  implemented  as  multiple  pan¬ 
els  on  a  large  display,  and  some  were  implemented  as  alternate  user  displays.  Also,  lessons 
learned  from  the  Dante  n  mission  to  an  Alaskan  volcano  [Bares  97]  included  the  importance  of 
graphical  displays  of  telemetry  in  a  multi-page  format.  There  is  also  demonstrated  value  in  the 
use  of  attitude  and  heading  reference  displays  similar  to  those  used  in  heads-up  displays  of  air¬ 
craft  [Hainsworth  00].  Definitions  of  the  four  modes  for  the  purposes  of  this  discussion  follow: 

Sensor/status  mode  provides  all  critical  information  about  the  robot’s  internal  state  and  its 
external  sensors.  It  may  also  be  called  telemetry  mode,  especially  in  the  case  of  machines  with 
little  or  no  autonomous  capability.  It  is  the  mode  typically  used  to  assess  the  operational  state  of 
the  robot  and  its  ability  to  proceed. 

Command  mode  provides  an  interface  tailored  to  issue  commands  to  the  robot,  perhaps  al¬ 
lowing  the  user  to  drive  it  with  a  virtual  joystick  or  with  cursor  keys,  to  actuate  mission- specific 
devices,  to  change  default  speeds,  gains,  and  sensitivities,  etc. 

Robot  perspective  mode  is  usually  dominated  by  one  or  more  camera  views  from  the  robot’s 
current  position,  but  may  include  any  environmental  perceptions  that  enhance  the  general  impres¬ 
sion  of  telepresence.  Remote  driving  is  often  conducted  from  this  display,  since  it  is  intuitively 
the  most  comfortable  for  typical  operators. 

Map  mode  provides  a  top-down  perspective  of  the  robot’s  local  environment,  placing  it  in 
context  with  known  (or  hypothesized)  environmental  features,  geographical  coordinates,  other 
robots,  etc.  Often,  multiple  levels  of  zooming  are  provided.  This  mode  is  often  required  for 
global  navigation. 

In  the  best  implementations,  there  is  modal  overlap,  and  it  is  therefore  possible  to  make  use 
of  some  of  the  same  functionality  in  different  modes,  but  the  rationale  for  the  different  modes 
usually  includes  a)  there  is  insufficient  space  in  the  display  to  provide  all  interfaces  at  once  and 
b)  even  if  there  were  enough  space,  it  would  be  overwhelming  to  the  operator. 

In  a  TMR  OCU,  it  is  especially  desirable  not  to  present  too  much  information,  so  the  multi¬ 
modal  model  is  appropriate,  and  the  typical  modal  interface  will  generally  be  even  less  informa- 


35 


tion-rich  than  full-fledged  GUI  interfaces.  With  creative  use  of  some  of  the  alternate  input  de¬ 
vices  (other  than  visual  displays),  it  becomes  possible  to  convey  some  additional  data,  including 
that  of  modes  other  than  the  active  mode.  And  if  the  primary  visual  display  is  effectively  trans¬ 
parent,  some  of  the  modal  information  can  be  displayed  as  non-distractive  AR  data. 

Several  opportunities  exist  for  the  effective  use  of  AR  displays.  During  manual  teleopera¬ 
tion,  for  example,  the  effect  of  autonomous  behaviors  (e.g.,  obstacle  avoidance)  can  be  masked 
from  the  actual  motor  control,  but  displayed  as  arrows  on  the  “driving  mode”  display  (e.g.,  as 
arrows  away  from  obstacles).  Similarly  attractive  forces  can  be  shown  as  well,  as  in  the  phero¬ 
mone  robotics  displays  of  Payton  [Payton  00] .  Also,  research  at  the  University  of  Maryland  has 
shown  that  there  is  great  value  in  showing  the  predicted  effect  of  operator  commands,  especially 
when  there  is  high  latency  in  communication  [Lane  00b] . 

With  respect  to  the  design  of  the  OCU  interface,  much  can  be  learned  from  the  efforts  at 
Carnegie  Mellon  in  the  area  of  collaborative  control,  their  description  of  joint  human-robot  op¬ 
eration,  in  which  the  robot  may  function  autonomously  and  make  requests  of  the  human  operator 
[Fong  99].  The  Georgia  Tech  MissionLab  system  supports  the  notion  of  teleautonomous  control, 
in  which  human  control  can  be  superimposed  upon  robotic  behaviors,  and  several  interface  con¬ 
cepts  have  been  developed  to  support  this  modality. 

2.6  OCU  System  Design 

From  the  standpoint  of  hypothesizing  an  integrated  system  design  that  would  be  appropriate 
for  TMR  applications,  it  is  proposed  that  a  multimodal  interface  (as  discussed  in  the  previous 
section)  be  implemented  on  a  hardware  suite  consisting  of  the  following  major  items: 

•  Wearable  computer, 

•  See-through  heads-up  display, 

•  Gesture-sensitive  glove  on  dominant  hand, 

•  Fingertip  contact-sensitive  glove  on  secondary  hand, 

•  Direct-contact  “joystick-type”  device  sewn  into  uniform  at  mid-thigh, 

•  Helmet-integrated  microphone  and  audio  feedback. 

Key  points  to  this  approach  include: 

•  Microphone  is  used  for  person-to-person  communication  only,  not  voice  recogni¬ 
tion, 

•  Attention-getting  “operator  interrupts”  provided  simultaneously  by  audio  alarms  and 
flashing  visual  indicators, 

•  Navigation  of  multimodal  visual  screens,  video  feeds,  etc.,  strictly  by  fingertip 
commands  of  secondary  hand, 

•  Use  of  dominant  hand  only  as  required  for  critical  positioning  and  gesturing,  includ¬ 
ing  glove  commands  or  joystick  control  (by  dropping  hand  to  thigh,  either  sitting  or 
standing). 
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Much  of  the  rationale  for  this  approach  has  been  provided  earlier  in  the  discussions  of  the 
various  devices  and  previous  user  interfaces.  Central  to  the  design  philosophy  is  the  fact  that  the 
operator  of  the  proposed  system  can  perform  a  variety  of  control  operations  without  requiring  use 
of  his  dominant  hand  or  losing  sight  of  the  immediate  surroundings. 

Specifically,  a  reasonable  approach  would  be  to  allow  the  operator  to  select  between  four 
different  operational  modes  by  “double-tapping”  his  thumb  against  one  of  the  four  fingers  on  the 
same  (non-dominant)  hand.  The  operational  modes  correspond  to  those  described  earlier,  and 
many  variations  are  possible,  but  for  example: 

•  Index  finger  -  command  mode 

•  Middle  finger  -  sensor  display  mode 

•  Ring  finger  -  map  mode 

•  Little  finger  -  immersive  robot  view  mode 

This  particular  choice  of  displays  might  be  designed  to  be  increasingly  immersive,  where  the 
command  mode  and  sensor  display  mode  simply  superimpose  a  few  gauges  and  indicators  over 
the  operator’s  normal  visual  field,  while  allowing  minimal  screen  navigation  and  user  input  with 
“single-taps”  of  the  thumb  against  the  four  fingers. 

The  dominant  hand  could  be  dropped  to  the  thigh-mounted  “joystick”  for  finer  control  of 
user  input  in  these  screens,  but  this  feature  would  primarily  be  used  for  driving  in  the  map  mode 
(which  takes  up  a  bit  more  of  the  visual  field),  and  the  immersive  “robot’ s-eye”  view  mode 
(which  ideally  is  seldom  used).  Similarly,  the  gesture  recognition  glove  on  the  dominant  hand 
could  be  used  for  additional  input  functions  in  these  modes,  under  the  same  assumption  that  the 
operator  would  not  have  entered  these  modes  unless  he  temporarily  had  use  of  both  hands  for 
robot  operation. 
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Figure  34:  Key  devices  for  the  implementation  of  the  proposed  approach,  including  the 
contact  sensitive  PinchGlove,  the  gesture  sensing  DataGlove,  the  low-profile  CyberPuck  for 
mounting  on  the  thigh  as  a  joystick,  and  the  virtual  retinal  display. 


The  actual  hardware  used  to  implement  (Figure  34)  this  could  consist  of 

•  any  of  a  variety  of  wearable  computers,  including  a  subset  of  the  Land  Warrior  sys¬ 
tem, 

•  a  PinchGlove  for  the  non-dominant  hand, 

•  a  DataGlove  for  the  dominant  hand, 

•  a  CyberPuck  for  attachment  to  the  uniform  at  the  front  of  the  thigh,  and 

•  the  Virtual  Retinal  display. 

Backups  to  all  of  these  devices  are  commercially  available,  but  the  largest  risk  is  probably  in 
the  display.  Both  the  MicroOptical  and  TekGear  displays  are  viable  alternative  to  virtual  retina 
technology.  All  of  these  devices  require  ruggedization,  but  all  are  suitable  for  experimentation  in 
their  current  form.  As  noted  earlier,  comms  gear  being  developed  for  Land  Warrior  and  other 
programs  should  be  integrated  where  possible. 
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3.  HUMAN  FACTORS  AND  TMR 


3.1  Human  Factors  Introduction 

In  designing  a  TMR  system  capable  military  use,  we  first  revisit  the  principles  of  teleopera¬ 
tion  and  study  how  this  has  evolved  into  telepresence.  We  can  learn  from  the  history  of  this  area 
about  human  factors  issues  that  affect  design,  operator  selection  and  training. 

[Draper  95]  defines  a  teleoperator  as  a  "general-purpose,  dexterous,  man-machine  system 
that  augments  man  by  projecting  his  manipulatory  and  pedipulatory  capabilities  across  distance 
and  through  physical  barriers  into  hostile  environments."  He  cites  several  human  factors  chal¬ 
lenges  in  teleoperation  which  are  likely  germane  to  TMR: 

•  Human  role:  what  level  of  control  is  chosen  (supervisory,  shared,  traded,  etc.) 

•  Design  of  feedback  systems  (viewing,  force) 

•  Pace  of  control  (user,  machine,  or  non-real-time); 

•  Role  of  feedforward 

•  Measuring  performance. 

An  interesting  quote  appears  in  [Draper  93]: 

In  teleoperation,  the  [human-machine]  interaction  ...  is  more  than  merely  an  exchange  of 
information  -  energetic  interactions  are  as  least  as  important. 

One  of  this  things  this  report  focuses  on  is  how  to  achieve  more  than  just  information  exchange 
and  to  be  able  to  tap  into  the  user’s  perceptual,  cognitive,  and  motor  channels  more  effectively 
than  with  mere  information  presentation. 

3.1.1  Organization  of  Control 

Sheridan’s  work  is  considered  central  to  telerobotics  research.  Figure  35  denotes  the  many  dif¬ 
ferent  aspects  of  control  feasible,  ranging  from  manual  to  supervisory  to  fully  automatic  (autono¬ 
mous)  control. 


Figure  35;  Levels  of  Control  for  Teleoperation  through  Autonomy  (from  [Sheridan92]) 
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3.1.2  Telepresence  and  Cognition 


[Draper  et  al  1998]  cite  studies  that  show  that  the  shape  of  the  robot  controller  is  not  impor¬ 
tant  during  teleoperation  but  spatial  correspondence  is.  The  artifact  must  be  easy  to  use  and  pre¬ 
dictable.  Other  studies  have  shown  that  displaying  force  information  via  audio  can  produce  better 
results  than  with  haptic  displays.  They  use  the  definition  of  telepresence  as  "a  mental  state  in 
which  a  user  feels  physically  present  within  the  computer  mediated  environment.".  Sheridan  re¬ 
fers  to  telepresence  as  the  loss  of  awareness  of  the  actual  user’s  surroundings.  Table  1  shows  the 
many  different  definitions  and  performance  factors  associated  with  telepresence,  while  Table  2 
lists  several  psychological  models  developed  for  telepresence. 


Approach  Nature  of  Telepresence 

Akin  et  al.  A  feeling  of  actual  presence 

(1983)  at  the  work  site 


Causes  Relationship  to 

_ Performance 

1.  Manipulator  dexterity  Telepresence  improves 

2.  Feedback  scope  performance 
and  Fidelity 


Sheridan  (1992 

User  feels  physically  present 

a,  b;  1996) 

at  the  remote  site;  compelling 
illusion;  subjective  sensation 

Steuer  (1992) 

The  sense  of  being  in  an 
environment;  the  experience 
of  presence  In  an  environ¬ 
ment  by  means  of  a  commu¬ 
nication  medium 

Zeltzer  (1992) 

Sense  of  being  in  and  of  the 
world 

Slater  and  Usoh 

The  (suspension  of  dis-)  be- 

(1993,  1994) 

lief  that  they  are  in  a  world 
other  than  where  their  real 
bodies  are  located 

Witmer  and 

subjective  experience  of  be- 

Singer  (1994) 

ing  in  one  place  when  physi¬ 
cally  in  another;  subjective 
sensation,  much  like  'mental 
workload';  a  mental  mani¬ 
festation 

Schloerb  (1995) 

The  person  perceives  that  he 
or  she  is  physically  present  in 
a  computer-mediated  envi¬ 
ronment 

Muhibach, 

Sense  of  sharing  space  with 

Becker,  and 
Prussod  ( 1 995) 

distant  interlocutors 

1.  Sensory  fidelity 

Telepresence  improves 

2.  Sensory  control 

3.  Manipulability 

performance 

1.  Vividness 

Telepresence  improves 

2.  Interactivity 

performance 

1, 

Autonomy 

Telepresence  may  im- 

2. 

Interaction 

prove  performance,  but 

3. 

Presence 

may  make  tasks  more 
difficult  and  fatiguing 

1. 

External  factors 

Telepresence  improves 

2. 

Internal  factors 

performance 

1. 

Control  factors 

No  clear  relationship 

2. 

sensory  factors 

3. 

distraction  factors 

4. 

realism  factors 

1. 

Information  flow 

Performance  must 

2. 

Ability  to  manipulate 

reach  some  minimum 

computer-mediated 

level  before 

environment 

telepresence  can  occur; 
relationship  not  estab¬ 
lished  beyond  that 

1. 

Spatial  presence 

Performance  improves 

2. 

Communicative 

presence 

telepresence 

Table  1;  Performance  factors  in  telepresence  (from  [Draper  et  al  98]) 


Approach 

Behavioral 

Cybernetics 

Flow 


Distal  Attribu¬ 
tion 


Nature  of  Telepresence 
Relationship  between 
feedback  and  feed-forward 
(not  experiential) 
Concentration  on  an  ac¬ 
tivity  to  exclusion  of  dis¬ 
tracting  stimuli 
a  compelling  impression  of 
being  at  the  location  oc¬ 
cupied  by  the  slave  device 


Situation  Maximization  of  SA  in  the 

Awareness  computer-mediated  envi¬ 

ronment  accompanied  by 
the  loss  of  SA  for  the  local 
_ environment 


_ Causes _ 

Degraded  by  temporal, 
spatial,  and  filtering  per¬ 
turbations 

Focus  on  task;  match 
between  task  require¬ 
ments  and  user  abilities 
Relationship  between 
sensory  inputs  and  nerv¬ 
ous  system  commands 
necessary  to  respond; 
moderated  by  quality  of 
afference-efference  link¬ 
ages 

Focusing  attention  on  the 
computer-mediated  envi¬ 
ronment 


Impact  on  Performance 
Performance  improves  with 
cybernetic  telepresence 

None 


Performance  improves  with 
telepresence 


Depends  on  importance  of 
local  SA 


Table  2:  Psychological  Approaches  to  Telepresence  (from  [Draper  et  al  98]) 
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Tele-existence  appears  to  be  synonymous  with  telepresence  and  is  defined  by  [Halme  and 
Suomela  2000]  as  "the  situation  where  the  main  senses  of  operator,  like  sight  and  hearing,  are 
transferred  to  the  remote  place  by  telecommunication  means  so  that  he/she  have  the  ’feeling’  of 
presence".  They  conducted  experiments  using  an  ATV  with  an  operator  in  a  stereo-head  mounted 
display,  with  head  tracking  audio  capabilities.  Five  scenarios  were  tested:  driving  in  a  corridor,  in 
unknown  terrain,  for  loading,  maneuvering,  and  fast  driving.  A  small  subject  pool  (5  men)  was 
used.  Principal  results  included  the  facts  that  training  improved  performance,  and  that  loading 
was  the  most  difficult  task.  System  differences  in  display  made  little  difference  after  training  on  a 
repeatable  task.  In  a  different  experiment  it  was  noted  that  servo  cameras  controlled  by  head 
movements  were  rejected  as  very  bad  by  all  drivers,  both  subjectively  and  objectively.  This  may 
have  been  due  to  the  position  of  the  cameras  that  were  located  where  a  driver  normally  sits,  in¬ 
stead  of  along  the  centerline  of  the  vehicle.  Camera  placement  appears  to  be  a  very  important  fac¬ 
tor  for  telepresence  systems.  Stereovision  did  help  in  the  loading  task.  The  head  mounted  display 
was  also  uncomfortable  and  induced  nausea  in  the  drivers.  The  conclusion  was  that  a  monitor 
was  better  than  the  technology  they  introduced. 

Another  study  in  telepresence  [Ballou  2000]  looked  at  remotely  operated  submersible  vehi¬ 
cles.  Again  a  very  small  data  set  was  used  that  is  inadequate  to  draw  conclusions.  Observations 
suggest  however  that  here  as  well,  operators  performed  better  with  a  panel  display  than  a  head 
mounted  one,  although  the  reasons  for  this  are  unclear.  It  was  also  noted  that  novice  users  tend  to 
lock  their  head  in  position  when  active  coupling  of  head  position  to  the  camera  is  present. 

[Simsarian  2000]  studied  user  interfaces  for  teleautonomous  control  in  the  context  of  virtual 
reality,  i.e.,  where  a  well-defined  model  exists  for  the  environment  in  which  the  robot  operates. 
While  this  is  an  interesting  premise,  unfortunately  it  holds  little  utility  in  battlefield  scenarios  as  a 
priori  models  of  the  world  are  generally  not  available  at  the  level  of  detail  required  nor  do  they 
necessarily  correlate  with  the  realities  of  the  moment. 

It  is  worth  observing  that  some  recognize  the  importance  of  trying  to  quantity  the  notion  of 
presence  in  systems  such  as  these.  [Prothero  et  al]  describes  vection  which  refers  to  a  "sense  of 
self-motion  induced  by  visual  cues".  They  provide  insights  into  how  metrics  for  certain  classes  of 
vection  can  be  measured.  Presence  is  less  well  defined  here,  but  they  advocate  a  measure  referred 
to  as  simulation  fidelity  in  terms  of  "the  ability  of  a  virtual  environment  to  induce  a  change  in 
perception  of  gravity-referenced  eye  level.  The  latest  references  refer  to  the  beginning  of  experi¬ 
ments  to  determine  the  usefulness  of  metrics  such  as  these. 

3.2  Sensory  Feedback 

This  section  surveys  the  impact  various  sensor  modalities  have  in  teleoperation  from  a  human 
factors  perspective;  i.e.,  what  works  and  what  doesn’t,  especially  as  related  to  TMR  issues. 

3.2.1  Sensory  Characteristics  of  Humans 

[Corliss  and  Johnsen  1968]  present  in  Tables  3-5  a  summary  of  the  sensory  channels,  and  their 
limits,  by  which  a  human  operator  can  be  engaged  with  a  teleoperator.  No  doubt  gaps  in  these 
tables  can  be  closed  and  refinements  made  due  to  human  studies  since  their  compilation,  but  they 
do  provide  useful  information  that  can  be  applied  to  interface  design. 
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Intensity  Range 

Intensity  Disi' 

riminaflifin 

Sense 

SinuUesI  Detec  lable 

Higlicsl  Praelkral 

RclatNe 

Ab&DtUte 

Vision 

2.2  «o  S.7  K  10 

Roughly,  the  brightness  of 
.snow  in  the  midday  sun,  or 
about  10^  times  the 

Ihreshnld  intensity 

With  white  light.  Ihom  are 
about  570  dlscrlmlnablc 
intensity  differences  in  a 
practical  range. 

With  white  liglit,  3  to  5 
absolutely  identiftable 
uilciisiUes  in  a  luiigv  of 

0.1  to  SO  millilomharts. 

Auclliion 

1  X  lU  erKs/cni^ 

Roughly,  the  intensity  of  the 
sound  piodueed  hy  a  jet 
plane  with  afterburner  or 
about  1 Q*  ^  times  the 
ihieshuld  inicnsily 

At  a  frequency  of  2.000  cps. 
iIjcic  die  uppuixi  It  lately 

325  dUcriminablc  intensity 
differences. 

With  purr,  tones  ahnut  3 
to  5  identlilable  steps. 

Mechanical 

vibrauon 

L'or  a  small  stimulator  on  the 
fingertip,  average  amplitudes 
of  0.00025  inin  can  be 
detected. 

Varies  witli  sue  of  atiinulutor» 
portion  of  hcttly  stimiilaled 
and  indKiduaL  Pain  Is 
usually  eneuuntcred  about 

40  db  above  threshold 

In  the  chest  region  a  broad 
contact  vibrator  with  ampli¬ 
tude  limlLs  between  0.05  mm 
and  0.5  mm  provides  15  dis* 
criminablc  amplitudes 

3  Co  5  steps 

Touch 

pressure 

Varies  consIdeniDly  with  body 
areas  stimulated  and  the 
type  of  stimulator.  Some 
representative  values: 

Uoll  of  thumb- 0.026  ci|;s 
l-'ingcrtips  0.037  to 
l.UyU  ergs 

Alin  0.032  tu 

0. 1 1 3  orgs 

Pain  threshold 

Vanes  enormously  lor  aiea 
iiicasurcd.  duration  of 
stimulus  contact  and  interval 
htHwetm  pnrsimlalion  of 
standard  and  comparison 
stimuli 

Unknown 

Kincs  thesis 

Joint  movements  of  0.  J 
degree  to  0.7  dcgicc  ul  a 
rate  of  10  dc|^niin  can  be 

delected.  (rt;nerally.  the 
laigcf  juints  arc  tlie 
must  sensitive 

Unknown 

No  ifvailahle 

No  data  availablt.* 

Anirular 

accclor- 

ution 

Depen  (icnl  on  the  type  of 
indicator  used 

1.  Skin  and  muscle 
senses  1  cleg/scc^ 

2.  Nystagmic  eye  move¬ 
ments  1  dcg/sec^ 

3.  Oculogyral  LUusio-n 

U.  1 2 

C 'nconsciuu&ncsb.  ur  “black- 
oiil"  occurs  tor  positive 
“g**  foiccs  uf  50  lu  8  g 
tasting  1  second  or  more 

Negative  forces  of  3  to  4.5  g 
cause  mental  confusion, 
“reel-vision"  and  extreme 
Itcaclachcs  la.sting  some¬ 
times  for  hours  fol¬ 
lowing  stimulation 

No  data  uvuiluble 

No  data  available 

Linear 

eiadon 

In  aircraft  — 0.02  g  for 
acoeli'cativtf  forces  and 

0.08  g  for  deeelerative 
forces 

I  'or  forces  acting  in  the 
direction  of  the  long 
axis  ul  the  body,  the  same 
liini UilitiiiN  as  fcir  angular 
acceleration  apply. 

No  dau  available 

No  data  available 

*A«luplcd  Iroin  Kef.  5S. 


Table  3.  Human  Sensory  channels  (from  [Corliss  et  al  68]) 
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Wavelength  or  Frequency  Range 

Wavelength  or  l-rcquency  Discrimination 

Sense 

Lowest 

Hipest 

Relative 

Absolute 

Vision 

(hue) 

300  nui 

1.500  mu 

At  medium  intensities  there 
are  about  128  discriininable 
hues  in  the  spectrum 

12  to  13  hues 

Interrupted 
white  light 

Unlimited 

At  moderate  intensities  and 
with  a  duty  cycle  of  0.5, 
white  li^t  fuses  at  about 

50  interruptions  per 
second 

At  moderate  intensities  and 
with  a  duty  cycle  of  0.5,  it 
is  possible  to  distinguish 

375  separate  rates  of 
interruption  in  the  range 
of  1  to  45  interruptions 
per  second 

No  greater  than  5  or  6 
interruption  rates  can  be 
positively  identified  on 
an  absolute  ba.sis. 

Audition 
(pure  tones) 

20  cps 

20,000  cps 

Between  20  cps  and  20,000 
cps  at  60  db  loudness,  there 
arc  approximately  1,800 
di.scriminablc  steps 

4  to  5  tones 

Interrupted 
white  noise 

Unlimited 

At  moderate  intensities 
and  with  a  duty  cycle 
of  0.5,  interrupted 
w  hite  noise  fuses  at 
about  2,000  interrup¬ 
tions  per  second. 

At  moderate  intensities  and 
with  a  duty  cycle  of  0.5. 
it  is  possible  to 
distinguish  460  separate 
interruption  rates  in  the 
range  of  1  to  45  interrup¬ 
tions  per  second. 

Unknown 

Mechanical 

vibration 

Unlimited 

Unknown,  but  reported  to 
be  as  high  as  10,000  cps 
witli  high  intensity  stimu¬ 
lation. 

Between  1  and  320  cps,  there 
arc  180  di.scriminabte 
frequency  steps. 

Unknown 

Table  4:  Discriminatory  capabilities  of  Human  Senses  (from  [Corliss  et  al  68]) 


successive 
stimuli 
Reaction 
time  for 
simple 
muscular 
movement 


Parameter 

Vision 

Audition 

Sufficient 

Uglit-radiatcd  electro¬ 

Sound-vibratory  energy, 

stimulus 

magnetic  energy  in  the 
wavelengths  from  400  to 
700  (violet  to  red) 

usually  airborne 

20  cps  to  20,000  cps 

Spectral 

120  to  160  steps  in  wave¬ 

"“3  cps  (20  to  1000 

range 

length  (hue)  varying 

cps)  0.3  percent  (above 

from  1  to  20  m**. 

1000  cps) 

Dynamic- 

~90  db  (useful  range) 

140  db  (0  db  =  0.0002 

range 

for  rods  =  0.00001  mL 
to  0.004  mL;  cones  = 

0.004  ml.  to  10.000  mL 

dyne/cm^) 

Amplitude 

resolution 

Al 

contrast  ■  0.015 

0.5  db  (1000  cps  at 

20  db  or  above) 

[fl 

Acuity 

10  areminutes 

Temporal  ac'uity  (clicks) 

=»0.001  sec 

Response  for 

M).l  sec 

'"0.01  sec  (tone  bursts) 

rate  for 

^0.22  sec 


M).  1 9  sec 


Touch 


Tissue  displacement  by 
physical  means 
>0  to  <400  pulses 
per  second 
App5_ 

■pps  ‘ 


.0.10 


'*'30  db.  0.01  mm  to 
10  mm 


0.15 


Two-point  acuity  = 

0.1  mm  (tongue)  to 
50  mm  (back) 

Touches  sensed  as  discrete 
to  20/scc 


sec  (for  finger 
motion,  if  finger  is  the 
one  stimulated) 


Vestibular 


Accelerative  forces 

Linear  and  rotational 
accelerations 


Absolute  threshold 
=*0.2  /sec/scc 


M).  10  change  in 
acceleration 


'*'1  to  2  sec  nystagmus 
may  persist  to  2  min 
after  rapid  changes 
in  rotation 


Best  op¬ 
erating 
range 
Indica¬ 
tions  for 
use 


500  to  600  M  (green- 
yellow)  to  to  200 
foot-candles 

1.  Spatial  orientation 
required. 

2.  Spatial  .scanning 
or  search  required. 

3.  Simultaneous  com¬ 
parisons  required. 

4.  Multidimensional 
material  presented. 

5.  Hi{^  ambient  nwe 
levels. 


300  to  6000  cps 
40  to  80  db 

1.  Warning  or  emer¬ 
gency  signals. 

2.  Interruption  of 
attention  required. 

3.  Small  temporal  rela¬ 
tions  important. 

4.  Poor  ambient  lifting, 

5.  High  vibration  or 
g  forces  present 


1.  Conditions  un¬ 
favorable  for  both 
vision  and  audition. 

2.  Visual  and  auditory 
senses. 


1  g  acceleration  di¬ 
rected  head  to  foot 

1.  Gross  sensing  of 
acceleration  infor¬ 
mation. 


*Adapted  from  Ref.  S5. 

Table  5:  Human  sensory  characteristics  (from  [Corliss  et  al  68]) 
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3.2.2  Vibratory  Feedback 


Kontarinis  and  Howe  at  Harvard  University  investigated  how  effectively  vibration  can  convey 
information  in  teleoperation  environments.  They  categorize  relevant  tasks  in  three  categories: 

1)  Where  detection  of  vibrations  is  the  fundamental  goal  (i.e.,  detecting  a  vibration  on  a 
door) 

2)  Where  performance  can  be  enhanced  by  indicating  the  mechanical  state  of  the  hand- 
object  system  (e.g.,  indicating  when  contact  has  occurred  as  in  grasping  a  doorknob). 

3)  Tasks  where  vibration  information  regarding  the  environment  in  not  important,  but  vibra¬ 
tion  increases  the  subjective  feel  of  the  environment  to  the  user. 

They  used  a  teleoperated  hand  system  to  conduct  experiments  for  ball  bearing  inspection,  punc¬ 
turing,  and  a  peg-in-slot  task.  Results  indicated: 

•  Force  and  vibration  are  complementary  modalities,  and  vibration  in  conjunction  with 
force  is  most  useful. 

•  The  importance  of  high-frequency  feedback  is  task  related. 

•  Display  design  is  low  cost,  using  a  simple  loudspeaker.  More  expensive  tactile  feedback 
units  can  be  provided  with  pins  to  operator’s  fingertips. 

[Desombre  et  al  95]  physically  shook  operators  using  a  vibratory  platform  and  evaluated  the 
consequences  on  their  ability  to  perform  simple  decision-making  tasks.  The  relevance  of  this 
study  to  TMR  is  for  operators  who  may  be  doing  mission  specification  or  run-time  control  while 
on  the  move  in  a  vehicle.  This  is  a  form  of  stress  testing.  Bar  graphs  seemed  better  perceived 
than  polygonal  visualizations  under  these  conditions  in  certain  cases.  Overestimation  of  graphic 
shapes  tended  to  occur  which  was  not  found  in  bar  graphs  when  vibration  was  present.  Oddly 
subjects  time  to  perceive  bar  graphs  when  shaken  decreased  over  stable  states. 

3.2.3  Haptic/Tactile  Feedback 

Hoffman  at  the  University  of  Washington  observed  that  most  VR  systems  do  not  provide  force  or 
tactile  feedback.  He  used  a  mixed  reality  feedback  system,  with  both  VR  and  actual  physical 
components,  which  indicated  that  the  subjects  performed  more  effectively  in  this  environment 
than  in  pure  VR  without  tactile  feedback.  This  is  not  surprising  but  does  document  these  results. 
This  same  group  expanded  this  work  further,  showing  that  tasting  virtual  objects  enhanced  the 
realism  of  these  virtual  experiences  as  well.  Again  not  a  surprise,  but  it  does  formally  document 
this  aspect.  The  experiment  involved  touching  and  tasting  a  candy  bar  when  the  same  object  was 
seen  within  a  virtual  environment.  It  is  unclear  how  this  could  be  practically  implemented,  but  it 
is  of  note  that  it  would  be  of  value  in  enriching  the  overall  immersive  experience  to  the  subject. 

[Burdea  96]  studied  haptic  (force  and  touch)  feedback  for  VR  applications.  In  his  book,  he  pro¬ 
vides  data  on  spatiotemporal  resolution  of  various  body  regions  (Table  6).  He  also  discusses  the 
various  classes  of  skin  mechanoreceptors  (Merkel  disks,  Ruffini  Corpuscles,  Meissner  corpus¬ 
cles,  and  Pacinian  corpuscles)  as  well  as  thermoreceptors  and  nocireceptors  (pain).  This  was 
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translated  into  a  figure  depicting  human  finger  sensing  and  control  bandwidth  suitable  for  hand 
feedback  (Figure  36).  This  all  translates  into  a  set  of  interface  requirements  shown  in  Figure  37. 


TABLE  2.1  Average  Pressure  JND  as  a  Function  of  Contact  Area 


r  ^ 

Body  site 

— 

Contact  Area  (cm^) 

1.27 

5.06 

20.27  1 

Pressure  JND 

Elbow  (Volar) 

0.167 

0.062 

0.040 

Elbow  (Dorsal) 

0.113 

0.052 

0.033 

Wrist  (Dorsal) 

0.188 

0.044 

— 

Overall  Average  JND 

0.156 

0.053 

0.037 

Adapted  from  Tan  et  al.  (1994J 

Table  6:  Spatiotemporal  resolution  of  various  body  locations  (from  [Burdea  96]) 


I  ,(X)()  Hz 


5,00() — 10.000  H/.:  The  bandwidth  over  which  the  hu¬ 
man  finger  needs  to  sense  vibration  dunng  skillful 
manipulative  tasks. 


320  H/.:  The  bandwidth  beyond  which  the  human  fin¬ 
gers  cannot  discriminate  two  consecutive  force  input 
signals. 


X 

T 

A 


l(X)  Hz 


10  Hz  . 

» 


J 


I  Hz  . 


20 —  30  Hz:  The  minimum  bandwidth  with  which  the 
human  finger  demands  the  force  input  signals  to  be 
present  for  meaningful  perception. 


12 — 16  H/:  The  bandwidth  beyond  which  the  human 
fingers  cannot  correct  their  grasping  forces  if  the 
grasped  object  slips. 


8 —  12  H/:  The  bandwidth  beyond  which  the  human  fin¬ 
ger  cannot  correct  for  its  positional  disturbances. 


5 —  10  Hz:  The  maximum  bandwidth  with  which  the  hu¬ 
man  finger  can  aply  force  and  motion  commands 
comfortably  . 


I  —  2  Hz:  The  maximum  bandwidth  with  which  the  hu¬ 
man  finger  can  react  to  unexpected  force/position 
signals. 


Figure  2.14  Human  finger  sensing  and  control  bandwidth.  Adapted  from  Shimoga  (19921. 
®  fiditions  Hermes.  Reprinted  by  permission. 


Figure  36;  Human  finger  sensing  response  (from  [Burdea  96]) 
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Figure  37:  Tactile  Interface  Requirements  (From  [Burdea  96]) 


3.2.4  Auditory  feedback 

The  [National  Research  council  Report  1998]  provides  guidelines  when  to  use  auditory  presenta 
tion  over  visual  presentation  for  soldiers  (Table  7).  Table  8  further  shows  the  advantages  and  dis 
advantages  of  variants  of  auditory  displays  of  information  to  the  soldier. 


I'sc  auditory  presentation  it: 


1  The  message  IS  simple. 

2  The  message  is  shon 

The  message  will  not  he  referred 
to  later. 

4.  The  message  deals  with  events 
in  time. 

5.  The  message  calls  for  immediate 
action. 

6.  The  visual  system  of  the  person 
is  overburdened. 

7.  The  receiving  location  is  too  bright 

or  dark  adaptation  integrity  is  necessary  , 
K  The  person’s  job  requires  him  or 
her  to  move  about  continually. 


I'sc  visual  presentation  if: 


I  I  he  message  is  complex. 

2.  The  message  is  long. 

3.  The  message  will  be  referred  to  later. 

4  The  message  deals  with  location  in  space. 

^  I  he  message  docs  not  call  for  immediate 
action. 

6.  The  auditory  system  of  the  person 
is  overburdened. 

7.  The  receiving  loealion  is  too  noisy. 

5  I  he  person  s  job  allows  him  or  her  to 
remain  in  one  position. 


Source:  Dcathcrage  (1972:  Table  4-1 ), 

Table  7:  Auditory  versus  visual  presentation  (From  [NRC  98]) 
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Ad\  anlaj:c 

Disadvantage 

Monaural 

Inexpensive 

Single  earphone 

.Sounds  can  not  be  localized 
Reduced  signal  delection 
Reduced  speech  intelligibility 

Hiaiiral 

Small  ineiease  in  signal  dclcelion 

Headphones  recpiired 

Sicrco 

Ineieased  signal  deieetion 

Increased  speech  inlclligibiliiy 

Minimal  increase  in  compicxilv 

Headphone  leijuired 

Improved  signal  delection 

Improved  speech  inlclligibiliiy 

Signals  locali/able 

Multiple  channels  monitoring 

Waypoint  navigation 

Increased  complexity 
Headphones  required 

Table  8:  Relative  Merits  of  Auditory  Feedback  Modalities  (From  [NRC  98]) 

[Hollier  et  al  99]  discuss  a  range  of  presentation  methods  and  teehnologies  using  3-D  spatial 
sound  to  the  end-user,  but  unfortunately  they  do  not  include  any  formal  user  analysis  of  its  effi- 
eacy. 

3.2.5  Visual  imagery 

Work  by  [Prothero  and  Hoffman  95]  at  the  University  of  Washington  confirmed  what  appears  to 
be  the  obvious,  that  a  wider  field  of  view  does  indeed  increase  the  feeling  of  presence  in  a  human 
subjeet  in  immersive  environments.  38  subjeets  were  used  to  confirm  this  hypothesis.  Goggles 
were  also  shown  to  be  better  (foreground  occlusion)  than  a  background  occlusion  (projection 
onto  paper). 

[Sayers  et  al  95]  at  the  University  of  Washington  studied  visual  feedback  for  undersea  remote 
teleoperation.  A  heavily  constrained  communications  channel  required  the  use  of  automatic  win¬ 
dowing  and  subsampling  of  the  image.  This  study,  however,  focused  on  the  technieal  design  of 
the  system  with  seeming  little  regard  for  operator  capabilities. 

3.2.6  Improper  Feedback:  Motion  Sickness 

Researeh  at  the  University  of  Washington’s  Human  Interface  Technology  Laboratory  studied  the 
effects  of  motion  sickness  in  virtual  environments.  Clearly  TMR  does  not  want  to  induee  this 
phenomenon  in  its  users.  This  group  has  constructed  a  means  and  metrics  by  which  motion  sick¬ 
ness  can  be  evaluated.  It  appears  that  it  is  primarily  due  to  visual-inertial  sensory  conflict.  A  solu¬ 
tion  appears  to  be  maintaining  an  independent  visual  background  for  the  immersive  environment 
that  is  consistent  with  the  subject’s  inertial  eues.  Thus  if  a  TMR  operator  is  driving  while  operat¬ 
ing,  feedback  from  the  driven  vehicle  should  be  coupled  to  the  immersive  environment  for  back¬ 
ground  projection. 
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Preliminary  research  has  attested  to  the  feasibility  of  this  approach  using  a  head-mounted  display. 
Observations  include: 

1)  That  a  simple  stationary  background  grid  in  the  sky  portion  is  insufficient  to  reduce 
motion  sickness. 

2)  The  induced  motion  of  the  background  grid  did  not  appear  in  peripheral  vision. 

3)  A  simple  foreground  grid  did  assist  in  reducing  motion  sickness. 

3.3  Operator  Performance 

Sheridan  states  [Sheridan  and  Ferrell  1974]  two  relevant  tenets:  when  working  at  capacity  one 
can  increase  speed  only  at  the  expense  of  accuracy,  and  conversely;  some  parallel  processing  can 
be  carried  out,  but  to  do  several  tasks  will  require  switching  among  them. 

In  reference  [Sheridan  and  Ferrell  1974]  to  an  earlier  study  by  Merkel,  response  times  are  in¬ 
creased  when: 

1 .  The  stimulus  is  unexpected 

2.  There  are  many  possible  stimulus-response  pairs 

3.  The  stimulus  or  responses  are  unfamiliar 

4.  It  is  difficult  to  discriminate  between  stimuli  or  when  searching  or  scanning  is  required 
for  detection 

5.  The  response  is  related  to  stimulus  in  complex  manner 

6.  The  response  in  difficult  to  perform. 

7.  Visual  stimuli  responses  are  longer  than  auditory  which  are  longer  than  for  tactile 
These  trends  can  serve  as  targets  for  TMR  interface  and  training  design. 


3.3.1  Metrics 

It  is  interesting  to  note  that  in  the  early  work  in  teleoperation  system  design  [Corliss  and  Johnsen, 
1968,  Johnsen  and  Corliss,  1967]  virtually  no  effort  was  dedicated  to  human  factors  evaluation, 
as  evidenced  in  statements  such  as  "Assertions  regarding  the  relative  effectiveness  of  different 
teleoperators  often  imply  that  considerable  experimental  data  exist,  but  no  thorough  studies  have 
been  made"  [Johnsen  and  Corliss,  1967].  Unfortunately,  the  situation  appears  only  slightly  better 
today,  with  more  lip  service  than  actual  experimental  studies  having  been  conducted.  Most  hu¬ 
man  factors  studies  have  not  focused  on  direct  measurements  of  human  control  of  robots,  but 
rather  involve  other  forms  of  control  (e.g.,  air  traffic  controllers)  from  which  we  must  extrapo¬ 
late. 

There  are  exceptions  however.  In  discussions  with  John  Draper  at  Oak  Ridge  National  Laborato¬ 
ries,  one  of  the  main  users  of  teleoperators  over  the  past  several  decades,  he  alone  is  involved  in 
human  factors  related  work.  In  a  sense  it  is  better  that  there  does  exist  at  least  one  individual 
within  DOE  to  design  and  conduct  these  studies,  but  it  is  a  sad  commentary  still  regarding  how 
engineering  typically  regards  human  factors  issues. 
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The  most  commonly  used  metric  for  quantifying  performance  of  teleoperators  is  task  completion 
time  [Book  and  Love,  1999].  Other  measures  considered  include  are  typically  resolved  into  total 
task  time,  time  effectiveness  ratio  (time  relative  to  if  the  task  was  performed  directly  by  the  hu¬ 
man),  and  unit  task  time  (related  to  performance  of  sub  tasks  of  the  overall  task).  Operator  fa¬ 
tigue,  success  ration  and  subjective  user  satisfaction  are  also  common  measures,  although  often 
difficult  to  evaluate  analytically.  Task-specific  metrics  are  also  created  on  an  ad  hoc  basis,  and 
may  in  this  context  deal  with  detectability  issues. 

So-called  information-based  performance  measures  relate  the  rate  of  information  that  is  transmit¬ 
ted  through  the  system  to  or  from  the  operator.  While  the  definition  of  manual  control  is  self- 
evident,  supervisory  control  refers  to  a  human  operator(s)  intermittently  or  continuously  pro¬ 
gramming  a  robot  that  itself  is  operating  autonomously.  The  operator  also  continuously  receives 
information  from  the  robot.  In  a  fully  autonomous  system,  the  human  operator  is  rather  an  ob¬ 
server  than  a  controller. 

3.3.2  Operator  Characteristics  in  Supervisory  Control 

Moray  outlines  the  general  characteristics  from  the  human  end  of  supervisory  control: 

1 .  Scheduling  what  to  do 

2.  Sampling  a  display  or  other  data  source 

3.  Acquiring  data  through  sensing 

4.  Combining  information  from  past  and  present  sources 

5.  Decision-making 

6.  Diagnosing  the  system  state 

7.  Executing  the  appropriate  action 

8.  Allocating  control  between  human  and  computer 

The  advantages  of  this  approach  over  pure  teleoperation  also  become  apparent: 

1 .  Improved  performance  over  operator  alone 

2.  Once  taught,  more  efficient  performance 

3.  Enables  tasks  operator  may  not  be  able  to  do  due  to  remoteness  (e.g.,  signal  delays) 

4.  Reduces  operator  workload 

5.  Improves  operators  task  planning  through  predictive  displays 

6.  Help  operator  to  monitor,  detect  and  diagnose  system  failures 

7.  Provides  failure  actions  when  operator  responses  would  be  too  slow 

8.  Makes  direct  control  easier  by  better  display  and  control  aids 

9.  Reduces  costs,  saves  lives  by  removing  operator  from  hazardous  environment. 

Eigure  38  shows  how  control  of  navigational  processes  can  be  affected  at  multiple  levels  un¬ 
der  supervisory  control.  Sheridan  further  indicates  that  human  skills  and  reasoning  proceed  at 
multiple  levels  as  well,  referring  to  Rasmussen’s  paradigm  (Eigure  39).  Of  particular  relevance 
for  the  robot  unit  concept  is  the  control  of  multiple  robots  concurrently  (Eigure  40). 
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Figure  38:  Supervisory  Control  (from  [Sheridan  93]) 


Mghargaaii 

vortMrt* 


Figure  39:  Multiple  Levels  of  Human  Reasoning  (from  [Sheridan  93]) 


Huin*ivlnl»r»c1ivB  System 
On  control  room  or  cockpit) 


multiptexad  signal  transmission 

(may  Involvs  barrdvndttr 
corortraints  or  tims  delays) 


Task-Interactive  System 
(remote  from  operator) 


Task  or  Controllad  Process 

(may  tie  continuous  process, 
vsiiicla.  robot,  or  etc.) 


Figure  40:  Control  of  multiple  computers/robots  concurrently  (from  [Sheridan  93]) 


Interestingly,  Sheridan’s  experimental  results  indicate  that  human  subjects  "approached  optimal 
behavior  ...  both  when  they  had  plenty  of  time  to  plan  and  when  their  workloads  were  reasonably 
heavy"  in  multitask  environments  [Sheridan  93].  There  was  an  observed  time,  as  workload  in¬ 
creased  when  the  subjects  switched  from  planning  ahead  to  putting  out  fires  and  shedding  tasks. 
This  clearly  points  to  a  need  for  effective  workload  management  on  a  robot  unit  operator. 
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Figure  41  shows  the  different  requirements  placed  on  the  operator  in  shared  control:  specifi¬ 
cally  including  the  abilities  to  plan,  teach,  or  intervene,  and  monitor  performance.  We  study  hu¬ 
man  factors  affecting  performance  of  these  duties  throughout  this  report. 


1  PLAN 


Figure  41:  Requirements  placed  on  Operator  (from  [Sheridan  93]) 

Sheridan  also  discusses  the  importance  of  human-attention  allocation  models  as  a  compo¬ 
nent  of  supervisory  control  systems.  Figure  42  shows  the  range  of  tasks  when  supervisory  control 
is  better  than  direct  teleoperation  relative  to  the  time  required  to  provide  the  capability  to  the  ro¬ 
bot. 


Figure  42:  Teleoperation  versus  Supervised  Control  (from  [Sheridan  93]) 
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3.3.3  Monitoring  Tasks 


[Thurman  and  Mitchell  95]  discuss  why  monitoring  is  difficult  via  operator  displays.  First  it 
is  a  boring  and  passive  task,  which  when  under  supervisory  control  does  not  require  active  en¬ 
gagement  of  the  operator.  Monitoring  is  also  an  unstructured,  ill-defined  task  for  complex  sys¬ 
tems.  Finally,  in  general  the  interfaces  operators  use  are  often  not  designed  for  monitoring. 
Sometimes  artificial  tasks  are  deliberately  added  to  displays  to  ensure  that  the  operator  remains 
engaged.  The  authors  advocate  a  specific  design  methodology  for  these  displays  based  on  task, 
operator  and  information-flow  analysis  which  has  been  shown  to  provide  better  fault  detection 
rates  among  other  things,  where  this  methodology  for  creating  operator  displays  for  highly  auto¬ 
mated  supervisory  control  systems  which  is  depicted  notionally  in  Figure  43.  It  starts  with  the 
necessary  operator  functions  and  proceeds  through  an  analysis  to  produce  the  necessary  informa¬ 
tion  at  the  right  time  in  the  display.  Although  it  was  tested  on  a  NASA  satellite  ground  control 
system,  it  has  relevance  for  TMR-related  display  design  and  should  be  considered  as  a  candidate 
methodology  for  the  design  of  such  a  system. 


Figure  43;  Design  Methodology  for  monitoring/control  interfaces 
(from  [Thurman  and  Mitchell  95]) 
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3.3.4  Driving/Navigation  Performance 


Specific  human  factor  problem  and  issues  related  to  driving  that  may  have  relevance  in  a  TMR 
robot  unit  include: 

•  Distraction  and  confusion  while  driving 

•  Reading  and  interpreting  maps,  text,  and  symbolic  displays 

•  Increased  hand  control  complexity 

•  Ignored  or  too-complex  warnings 

•  Excessive  dependence 

•  False  alarms 

•  Effective  heads-up  displays 

•  Situational  awareness  displays 

•  Automatic  control  of  speed/braking  or  steering  imposed  on  human  control 

•  Transients  in  human  to  fully  automated  platoon  control 

•  Variations  in  operators  due  to  age,  education,  etc. 


Work  by  [Eevison  and  Baron  1997],  studied  the  impact  of  task  difficulty  in  driving  a  remote  ve¬ 
hicle.  It  is  possible  that  these  results  may  potentially  extrapolate  to  multi-vehicle  remote  driving 
as  needed  for  TMR  Robot  unit  applications.  Table  9  summarizes  their  results. 


Perf  ; 

Performance  Trends 

1 

Model 

Experiment 

1 

Multitask  vs  single-  1 

Multitask 

Multitask 

task  ' 

driving  performance 

worse 

worse 

Increasing  difficulty 

-More  attention 

More  attention 

of  the  driving  task 

to  driving  task 

to  driving  task 

Driving 

Driving 

performance 

performance 

degraded 

degraded 

Less  attention 

No  significant 

Increasing  relative 

to  driving  task 

influence  on 

importance  of  the 

attention  or 

auxiliary  task 

Driving 

driving 

performance 

degraded 

performance 

Better 

Better 

Driver-controlled 

performance 

performance 

attention 

when 

when 

not  attending 

not  attending 

to  road 

to  road 

Better 

Periodic 

performance 

attention 

when 

attending 
to  road 

Table  9:  Driving  Performance  (from  Levison  and  Baron  97]) 
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An  earlier  experiment  by  Noy  summarized  in  [Levison  and  Baron  1997]  found  that: 

1 .  Driving  errors  were  larger  for  dual-tasks  than  single  task  conditions 

2.  Making  driving  harder  had  only  modest  impact  on  driving  performance,  a  larger  impact 
on  scan  behavior,  and  little  impact  on  secondary  task  performance 

3.  Making  the  difficulty  of  the  auxiliary  task  harder  did  not  affect  the  driving  and  scanning 
performance  significantly,  but  had  large  impact  on  auxiliary  task  performance. 

4.  Oddly,  driving  performance  was  slightly  better  when  subject  when  looking  at  auxiliary 
task  display  rather  than  road. 

[Draper  et  al  99]  cite  a  study  for  robotic  rover  navigation  tasks  conducted  in  simulation.  It  was 
interesting  to  note  that  the  more  immersive  the  environment  was,  the  faster  it  took  to  conduct  the 
navigational  task.  There  was  a  correlation  that  showed  immersive  environments  helped  for  this 
class  of  task  (which  was  not  always  the  case  for  others).  It  was  believed  that  the  ability  to  direct 
attention  assisted  here. 

[Peterson  et  al]  at  the  University  of  Washington  studied  the  effects  of  two  different  inter¬ 
faces  for  navigation.  One  was  a  joystick  controller  and  the  other  a  sophisticated  virtual  motion 
controller  that  senses  the  body  of  the  user  by  having  him  stand  on  it  (Figure  44).  Maneuvering 
performance  was  better  with  a  joystick  when  following  a  marked  route.  The  ability  to  form  a 
mental  map  (survey  knowledge)  and  alternate  route  planning  was  better  with  the  body  controller 
and  was  dependent  on  how  difficult  the  task  environment  was  (i.e.,  the  more  difficult  the  better 
the  body  controller  was).  It  was  postulated  that  the  additional  kinesthetic  and  vestibular  modali¬ 
ties  might  have  aided  sensory  integration  that  people  normally  expect  while  wayfinding  in  un¬ 
known  areas.  Route  learning  (replicating  the  same  route)  was  the  same  for  both  devices.  The  sub¬ 
jects  used  head  mounted  displays. 


Figure  44:  Experimental  Setup  (From  [Peterson  et  al]) 
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3.4  Human  Factors  and  Displays 


Problems  with  displays,  in  general,  include  information  overload,  display  inter-correlation  and 
redundancy;  and  the  keyhole  problem  (if  information  is  not  on  the  screen,  how  to  get  it?).  This 
section  focuses  both  on  general  issues  as  well  as  problems  and  trends  associated  with  specific 
display  types. 

3.4.1  Helmet  Mounted  Displays 

[Schrage  94]  evaluated  the  state  of  the  art  in  helmet-mounted  displays  (HMDs)  in  terms  of  hu¬ 
man  factors  for  aviation.  A  vendor  inventory  assessment  was  conducted.  The  human  factors 
study  was  related  to  the  anthropomorphic  physical  properties  of  the  display  system,  than  user 
perceptions,  were  they  were  ranked  for  adjustability,  range  of  head  movement  and  comfort.  Addi¬ 
tional  psychophysical  tests  were  performed  on  a  pool  of  4  subjects,  for  data  such  as  image  sharp¬ 
ness,  color  quality,  acuity,  and  image  quality.  These  were  not  evaluated,  however,  in  terms  of 
task  performance  unfortunately.  Therefore  we  cannot  draw  conclusions  from  this  study  as  to  how 
well  HMDs  will  or  will  not  work  in  a  TMR-related  task  environment. 

3.4.2  Heads-up  Displays 

Other  studies  have  pointed  out  difficulties  with  Heads-up  displays  (HUDs).  In  one  case  [Foyle  et 
al,  McCann  et  al],  pilots  took  2.5  seconds  longer  in  responding  to  runway  incursions  with  HUDs 
than  without  the  technology.  It  is  believed  this  is  a  consequence  of  attentional  tunneling  leading 
to  inefficient  processing  of  the  regular  out-of-the-window  scene.  Thus  the  use  of  superimposed 
symbology  in  HUDs  is  not  always  a  blessing.  Use  of  scene-linked  symbology,  i.e.,  enhancements 
of  existing  features,  apparently  assists  in  alleviating  this  psychological  phenomenon.  So  careful 
choices  should  be  made  on  what  and  how  data  are  represented  in  heads-up  displays.  The  authors 
recommend  removing  the  HUD  cues  that  distinguish  its  information  from  the  natural  world,  in 
particular:  color  (typically  saturated  green),  lack  of  perspective  (planar  instead),  and  differential 
motion  (it  does  not  move  with  the  world). 

The  National  Research  Councils  Report  on  Tactical  Displays  for  Soldiers  [1998]  recommends 
that  if  the  display  occludes  the  soldiers’  view,  then  hand-held  or  wrist  mounted  displays  should 
be  considered  as  an  alternative  to  helmet  mounted  displays.  The  loss  of  local  situational  aware¬ 
ness  may  be  unacceptable. 

3.4.3  Other  Display  considerations 

Studies  involving  displays  for  air  traffic  controllers  were  also  considered  as  relevant,  as  they  are 
used  to  improve  their  situational  awareness  [Johnston  et  al],  a  crucial  TMR  task.  These  SADs 
(situational  awareness  displays)  currently  rely  heavily  on  alphanumeric  information  requiring 
multiple  eye  fixations  and  complex  mental  transformations  to  generate  the  3D  representation  of 
the  world.  One  notion  is  the  use  of  perspective  displays  (as  advocated  for  HUDs  above).  Color 
was  also  shown  to  be  better  than  achromatic  displays  and  was  demonstrated  to  work  well  for 
managing  attention.  Using  various  spot  sizes  to  represent  planes  that  correlated  to  altitude  also 
provided  useful  information  regarding  altitude  versus  uniform  size  spots. 
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Another  potentially  related  study  for  teleoperation  involved  studies  in  support  for  pilots  on  taxi- 
ways  in  conditions  of  low  visibility.  We  have  just  seen  in  the  Singapore  airlines  disaster  that  such 
failure  can  lead  to  catastrophes.  In  certain  aircraft,  heads-up  displays  provide  symbology  to  the 
pilot.  A  recent  addition  is  the  electronic  moving-map  display  that,  using  GPS  technology,  depicts 
the  aircraft’s  location  in  real-time.  In  some  case,  route  guidance  is  provided  on  the  display  as 
well,  showing  where  the  plane  should  go  as  cleared  by  the  tower.  Pilots  are  able  to  move  at 
greater  speeds  and  make  fewer  navigation  errors  when  these  displays  are  available,  especially 
when  route  guidance  is  provided  [McCann  et  al].  The  main  problem  seems  to  lie  in  the  coordi¬ 
nate  system,  which  is  world-centered  instead  of  egocentric,  requiring  the  pilots  to  make  several 
cognitive  transformations  that  are  "effortful  and  time-consuming".  Work  is  ongoing  in  trying  to 
align  3D  perspective  displays  to  eliminate  these  problems.  The  3D  displays  did  result  in  slower 
taxi  speeds  despite  the  pilots’  preference  for  them,  evidently  due  to  fewer  look  ahead  cues  being 
available.  Variable  zoom  levels  were  also  presented  and  the  pilots  generally  preferred  the  highest 
zoom.  [Foyle  et  al  96]  describe  a  system  engineered  to  assist  pilots  based  on  human  factors  con¬ 
siderations  for  this  very  task,  including  scene-linked  HUD  symbology,  a  perspective  moving  map 
display,  and  3D  audio  ground  collision  and  warning  system.  The  audio  system  actually  provided 
stereo  feedback  as  to  the  direction  of  the  impending  collision. 

3.5  Mental  Workload 

Mental  workload,  according  to  [Sheridan  93]  consists  of  objective  factors  such  as  number  of 
tasks,  urgency,  and  cost  of  non-completion  of  task  on  time  or  correctly,  as  well  as  a  range  of  sub¬ 
jective  factors  and  environmental  variables  (workstation  display  geometry,  ambient  temperature, 
lighting,  etc.)  (Figure  45). 
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Figure  45:  Mental  Workload  Factors  (from  [Sheridan  93]) 
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Performance  tends  to  decrease  steeply  when  workload  becomes  too  high,  while  through  the 
normal  range  little  change  is  noticed.  It  is  worth  noting  that  if  the  operator  has  too  little  to  do, 
that  performance  declines  as  well  due  to  inattention  and  lack  of  vigilance  for  monitoring.  This  is 
illustrated  in  Figure  46.  Sheridan  provides  a  rating  system  for  workload  measurement  (Figure  47) 
and  a  decision  model  associated  with  it. 
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Figure  46:  Relation  of  Performance  to  Workload  (from  [Sheridan  93]) 
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Left;  ungle-dimensioiul  subjective  roentel  woritloed  scale  developed  by  Wierwille  and 
Casali  (1993)  and  patterned  after  the  Cooper^Harper  aircraft  handling  quality  scale.  Right; 
three-dimensional  subjective  mental  workload  scale,  developed  by  Reid  et  al.  (1981) 
following  Sheridan  and  Stmpaon  (1979). 


Figure  47:  Workload  Measurement  Rating  System  (from  [Sheridan  93]) 


Subsequent  research  by  [Draper  and  Blair  96]  studied  the  role  of  workload  and  flow  in  a 
similar  context.  In  this  study  highly  trained  operators  were  used.  They  employed  the  NASA  TLX 
workload  index  as  a  basis  for  quantifying  their  measurements  (Table  10).  The  dimensions  shown 
in  Table  11  measured  flow.  Similar  metrics  could  potentially  be  applied  to  TMR-related  tasks. 
This  led  to  their  development  of  a  cognitive  model  for  teleoperation,  which  addresses  the  role  of 
attention  (Figure  48).  It  would  be  interesting  to  see  how  this  model  stood  up  to  TMR  type  tasks. 
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Scale  Name 

Scale  Definition 

Mental 

How  much  mental  and  perceptual  ac- 

tivity  was  required? 

Physical 

How  much  physical  activity  was  re- 

quired? 

Temporal 

How  much  time  pressure  did  you  feel? 

Performance 

How  successful  do  you  think  you  were? 

Effort 

How  hard  did  you  have  to  work  (men- 

tally  and  physically)? 

Frustration 

How  discouraged  versus  gratified,  an- 

noyed  versus  content  did  you  feel 
during  the  task? 

Table  10:  NASA  Study  Dimensions  (from  [Draper  and  Blair  96]) 

Flow  Dimension  Flow  Deflnition 

Goals  Clarity  of  task-related  goals 

Attention  Strength  of  commitment  of  atten- 

tional  resources  to  task 
performance 

Challenge  Degree  to  which  the  task  was  a 

challenge  for  the  operator 

Control  Degree  to  which  the  operator  felt  in 

control  of  the  situation  during 
task  performance 

Gthers  Degree  to  which  operator  was  con¬ 

cerned  about  other  persons’ 
opinions  of  his  performance. 

Table  11:  Study  Dimensions  of  Flow  (from  [Draper  and  Blair  96]) 
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Figure  48:  Attention  Model  for  Teleoperation  (from  [Draper  and  Blair  96]) 
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[Wei  et  al  1995]  conducted  user  studies  in  operator  supervision  of  a  system  and  its  relationship  to 
workload.  The  came  to  several  interesting  conclusions: 

1 .  It  does  not  always  follow  that  with  more  automation  that  operator  performance  will  im¬ 
prove. 

2.  Optimal  balance  between  mental  load  and  system  performance  depends  on  both  task 
number  and  task  characteristics. 

The  overall  conclusion  is  that  the  functions  that  should  be  automated  in  a  system  depend  on 
both  what  enhances  system  performance  and  reduces  operator  workload  (improves  operator  per¬ 
formance). 

3.6  Human  error 

A  major  question  is  when  should  the  operator  intervene  within  autonomous  systems.  Human  er¬ 
rors  seem  to  be  most  apparent  during  intervention  [Sheridan  93].  Human  error  is  an  action  that 
fails  to  meet  some  implicit  or  explicit  standard  of  the  actor  or  of  an  observer  [Sheridan  93]. 
There  exist  several  taxonomies  for  this  sort  of  error,  most  notably  the  Senders-Moray  taxonomy. 
Errors  occur  in  the: 

1.  Environmental  context:  omission,  substitution,  unnecessary  repetition,  etc. 

2.  Cognitive/behavioral:  attention,  perception,  memory,  or  control 

3.  Bias:  anchoring,  overconfidence,  confirmation,  cognitive  lockup,  tunnel  vision 

4.  Behavioral  complexity:  simple  sensing,  detecting,  identifying  classifying  errors;  sequenc¬ 
ing  from  memory;  estimation  errors  in  responses;  reasoning  errors 

5.  Exogenous  (caused  by  actor)  or  endogenous  (caused  by  actor’s  environment) 

The  main  sources  of  error  [Sheridan  93]  include: 

•  Eack  of  feedback 

•  Capture  (inappropriate  conditioned  responses) 

•  Invalid  internal  models 

•  Hypothesis  verification  on  the  basis  of  inadequate  data 

•  Stress  and  perceptual  narrowing 

•  Risk  homeostasis  (constant  level  of  risk  maintained  even  with  safety  aids) 

•  Individual  error  proneness 

•  CNS  factors  (sleep,  drugs,  emotionality,  etc.) 

Error  factors  can  be  reduced  through  solid  design  practices,  training  incorporating  the  ability  to 
think  about  potential  errors,  and  suitable  environmental  conditions. 

Another  class  of  errors  that  operators  are  prone  to  when  dealing  with  complex  system  are  associ¬ 
ated  with  systems  that  function  in  various  modes.  This  leads  to  mode  error  and  ambiguity  [Dez- 
ani  et  al  98].  Mode  errors  occur  when  a  situation  is  incorrectly  recognized,  and  although  the  cor¬ 
rect  action  is  taken  for  the  perceived  situation,  it  results  in  an  error  due  to  the  misperception. 
Mode  ambiguity  is  a  bit  different.  This  occurs  when  the  operator  has  a  different  expectation  re¬ 
garding  the  outcome  of  the  undertaken  action  in  response  to  the  situation.  Obviously,  neither  of 
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theses  situations  is  desirable,  and  effort  must  be  made  to  reduce  or  eliminate  the  possibility  in  a 
TMR-style  system.  While  the  authors  discuss  methods  to  model  these  events,  unfortunately  no 
general  proactive  actions  are  prescribed. 

Dual-task  interference  is  another  type  of  error.  [Van  Selst]  notes  that  this  problem  occurs  even  in 
the  absence  of  the  need  for  a  response.  It  is  a  consequence  of  the  psychological  refractory  period 
effect,  which  provides  a  fundamental  limit  on  human  parallel  task  performance.  There  appears  to 
be  a  central  bottleneck  in  human  information  processing  associated  with  the  so-called  cognitive 
slack  associated  with  the  problem.  His  study  points  to  the  root  cause  being  related  to  stimulus 
classification  and  not  action  formulation.  There  is  extensive  literature  on  this  phenomenon,  and 
in  TMR  we  must  be  concerned  how  to  eliminate  parallel  situations  to  ensure  rapid  and  correct 
responses. 

Another  source  of  human  error  is  when  the  operator’s  mental  model  of  the  system  is  not  in 
agreement  with  the  system.  [Palmer]  studied  flight  operations  for  MD-88  and  DC-9s  and  noticed 
that  so-called  automation  surprise  occurred  when  using  the  autopilot  and  autothrottle  resulted  in 
altitude  deviations.  In  the  case  study,  the  automation  worked  correctly,  but  it  did  not  respond  as 
expected  by  the  pilots.  The  lesson  is  to  develop  designs  that  either  eliminate  unusual  automation 
modes  or  provide  better  displays  of  normal  but  unusual  modes.  Basically  a  "what  you  see  is  what 
you  get  display"  is  needed. 

3.6.1  Fatigue 

Managing  fatigue  is  another  important  area,  especially  for  combat  systems.  [Rosekind  et  al]  have 
studied  this  problem  for  NASA.  Fatigue  is  unavoidable  as  there  are  circadian  rhythms  that  force 
us  into  certain  patterns  of  sleep.  Sleepiness  has  been  shown  to  affect  waking  performance,  vigi¬ 
lance  and  mood  even  to  the  point  where  an  individual  is  unable  to  perform  at  all.  There  are  coun¬ 
termeasures  that  can  be  undertaken:  minimizing  sleep  loss,  having  good  sleep  habits,  providing  a 
comfortable  sleep  environment,  minimize  the  effects  of  food,  alcohol  and  exercise,  as  well  as 
manage  circadian  clock  (chronobiotics).  Social  interaction,  caffeine,  bright  light  and  melatonin 
all  can  be  used  to  ensure  alertness.  There  is  little  information  on  the  design  of  systems  to  manage 
this  affect,  but  operator  fatigue  management  should  play  a  role  to  ensure  reliable  performance  in 
TMR  systems. 

3.6.2  Trust 

[Wickens  et  al  98]  discuss  the  importance  of  trust  by  the  operator  in  the  automation  they  are  us¬ 
ing.  This  is  clearly  essential  for  acceptance  of  TMR-related  technology  as  well.  They  cite  7  main 
attributes  of  trust: 

1 .  Reliability 

2.  Robustness 

3.  Familiarity 

4.  Understandability 

5.  Explication  of  intention 

6.  Usefulness 

7.  Dependence  of  trusting  person  upon  automation. 
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If  any  of  these  criteria  are  neglected  in  the  design  of  a  TMR  system,  it  will  adversely  affect  the 
acceptance  by  the  end-user. 

There  is  a  danger  as  well  in  overtmst  and  complacency  as  well  as  mistrust  when  operators  rely  on 
complex  automation  more  than  they  should.  A  calibration  process  for  trust  in  the  operator  is  re¬ 
quired  through  the  construction  of  accurate  mental  models  by  training  and  through  the  participa¬ 
tion  in  automation  design  by  potential  operators. 

3.6.3  Vigilance  Decrement 

Human  performance  degrades  over  time  when  monitoring  systems  with  low  event  rates  -  this  is 
called  the  vigilance  decrement.  Providing  alerts  and  cues  when  significant  events  occur  help  re¬ 
duce  this  phenomenon.  Presenting  a  video  image  at  that  time  may  be  best  rather  than  continuous 
output.  The  operator  is  then  challenged  to  identify  what  and  where  the  event  is.  Mobile  sensor 
platforms  complicate  the  job  for  the  operator,  as  the  images  are  not  static  when  compared  to 
fixed  cameras.  Redundant  reports  from  multiple  overlapping  sensors  also  must  be  correlated. 
Work  at  NOSC  [Murray  1995]  conducted  studies  on  this  problem  with  focus  on  multiple  moving 
robotic  systems  but  in  indoor  simulation  only.  Of  note  is  the  comment: 

Results  of  this  study  provide  some  guidelines  for  predicting  human-machine  performance 
for  systems  involving  multiple,  autonomous  sensors.  The  rapid  increase  in  response  time 
for  even  the  modest  levels  of  manipulations  used  here  is  cause  for  concern  especially 
when  newer  systems  are  planned  with  large  numbers  of  sensors  and  are  designed  for  op¬ 
erations  in  cluttered  environment. 

Human  performance  may  be  rate  limiting  in  deploying  multiple  TMRs.  [Murray  95]  contends 
that  traditional  supervisory  control  (a  la  Sheridan)  does  not  hold  here.  He  advocates  information 
extraction  by  the  individual  sensors  ("keyholes")  into  a  coordinated  picture  that  could  then  possi¬ 
bly  make  the  task  more  tractable.  Also  adding  visual  cues  to  the  imagery  (overlays)  and  methods 
to  reduce  redundancy  from  sensor  overlap  might  also  help. 

3.6.4  Stress 

The  National  Research  Council  Report  cites  numerous  sources  of  stress  for  the  soldier  that  can 
have  impact  on  interpretation  of  display  performance.  These  include 

•  Physical  Sources 

o  Heat  and  Cold 
o  Noise 

o  Vibration  (see  Section  2.1) 
o  Extended  operations/Time  of  Day 

•  Operational  Sources 

o  Task 

o  Information  overload  or  underload 
o  Information  versus  disinformation  (trust) 
o  Physical  work 

These  factors  can  result  in  fatigue  or  performance  failure. 
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Several  specific  design  guidelines  for  soldier  displays  are  forwarded. 

1.  Rely  on  absolute  judgment.  Don’t  make  calls  on  variables  such  as  intensity  or  color  with 
more  than  5-7  levels. 

2.  Don’t  violate  expectations  based  on  previous  experience. 

3.  Present  the  same  information  in  multiple  formats  (voice  and  visual)  -  it  increases  under¬ 
standing. 

4.  Make  displays  represent  things  realistically. 

5.  Moving  displays  should  match  mental  model  of  users  expectations. 

6.  Ecological  design  should  encourage  display  to  maintain  correspondence  with  the  envi¬ 
ronment  it  represents. 

7.  Information  gathering  cost  should  be  minimized.  Reduce  eye  movements  from  one  loca¬ 
tion  to  another. 

8.  Use  multiple  sensory  resources  (sound  and  sight). 

9.  Provide  predictive  aids. 

10.  Place  required  information  in  environment  such  as  checklists. 

1 1.  Be  consistent  across  displays  and  with  user  habits. 

12.  Provide  graphic  displays  whenever  possible  as  opposed  to  textual. 

13.  Simplify  graphics  as  much  as  possible  reduces  workload  in  high-stress  environments. 

14.  Present  salient  info  in  center  of  display;  peripheral  information  is  neglected  in  high-stress 
situations. 

15.  Use  redundant  audio  and  visual  warnings  for  threat  location. 

16.  Reduce  data  entry  requirements  and  menus  to  bare  minimum  during  engagements 

17.  Use  a  small  and  understandable  icon  set. 


3.7  Human  Factors  in  Design 

Now  we  examine  how  human  factors  considerations  can  and  should  influence  the  design  of 
complex  systems  such  as  a  TMR  robot  unit  operator  control  unit. 

3.7.1  Design  Methodologies 

In  this  early  work,  [Corliss  et  al  68],  they  partition  the  design  task  into  the  following  allocations: 

•  Human  assignments:  Pattern  recognition,  target  identification,  new  exploration,  long-term 
memory,  trouble  shooting  and  emergency  operation,  planning,  interpreting  complex  data, 
inductive  thinking,  settings  goals  and  priorities  while  evaluating  results 

•  Machine  assignments:  Monitoring  multichannel  inputs,  boring  repetitious  tasks,  precise 
motions  and  force  applications,  high  speed  motions,  short-term  memory,  optimization, 
deductive  analysis,  computing,  and  monitoring 

They  arrived  at  this  division  using  an  algorithmic  systems  approach  to  partitioning  the  work  be¬ 
tween  man  and  machine.  The  following  design  activities  are  prescribed  in  sequence  and  could  be 
applied  to  a  TMR  robot  unit  as  well. 
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1 .  Hypothesize  the  potential  basic  role  of  man 

2.  Hypothesize  potential  complementary  and  support  role  of  man 

3.  Review  manned  system  solution  feasibility 

4.  Develop  a  preliminary  operator  concept 

5.  Analyze  personnel  support  requirements 

6.  Review  potential  operator  role  for  acceptance  and  reliability 

7.  Synthesize  optimal  operator  role 

8.  Establish  feasibility  of  man-rated  allocation 

9.  Develop  potential  man-rated  allocations 

10.  Review  allocation  potential  against  psychophysical  capacities 

1 1 .  Review  allocation  potential  against  system  or  function  constraints 

12.  Review  allocation  potential  against  human  reliability 

13.  Synthesize  man-rated  allocations 

A  related  design  approach  appears  in  [Johannsen  1997]  and  is  shown  in  Figure  49: 


Figure  49.  Johannsen’s  Design  Methodology  (from  [Johannsen  97]) 

In  another  study,  [Draper  93]  cites  three  different  ways  to  design  telemanipulator  systems: 

1)  Minimalist  (TV  and  joystick) 

2)  More-is  better-  approach  (full  spectrum  of  sensory  feedback) 

3)  Mission-oriented  approach:  acknowledges  design  tradeoffs  in  capability  and  cost 
seeking  efficient  solutions. 
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He  recommends  (as  do  we)  the  last  of  these  approaches  as  the  basis  for  design. 

[Draper  et  al  99]  further  compares  user-centered  versus  telepresence  design  approaches  (Table 
12).  This  helps  to  separate  the  hype  surrounding  immersive  environments  from  the  realities. 


tnfbnnatlon 


Respaiuea 


Attentian 


Fatigue/ease  of  use/ 
workload 


Training 


Table  1.  Compaxoon  and  Contkast  or  E)sscn  Appxoaches 


User-cenicral 


Immmivt/telepresenct 


Efficient  displays  promoting  situation 
awareness;  irtformation  may  be 
summarized  or  pre-processed  before 
presentation 


Etgonorrucal  controls  accepting 
inputs  from  modalities 


focus  attention  on  critical  parts  of  a 
task 

Interfaces  desigtxed  to  be  minimally 
fatiguing  and  for  ease  of  use; 
workload  is  an  important  concern  and 
the  task  rxtay  be  inodified  to  maintain 
acceptable  l^els  throughout  the 
rtusaion 

Some  traiiting  is  usually  assumed  to 
be  necessary  to  efficiently  operate 
interfaces 


Displays  that  reproduce  the  sensory 
milieu  at  the  remote  or  simulated  site 
and  eliminate  stimulation  from  the 
user's  physical  environinent  (all 
mformation  presented  may  itot  be 
relevant  to  developing  situation 
awareness  for  task  performance) 

Devices  that  digitize  hunun  actions 
(eg.,  walking)  or  the  results  of  human 
actions  (e.g.,  sound,  as  in  speech)  in  a 
tutural  way 

Controls  aitd  displays  are  designed  to 
re-create  the  environment,  usually  with 
no  attempt  to  shape  attentioruil 
processes 

Interfaces  dcsigrted  to  be  no  more 
fatiguing  thw  if  the  user  was 
physically  present  in  the  SE:  ease  of 
use  is  assumed  to  be  optimal  because 
of  ruturalness  atrd  transparency; 
workload  is  not  raised  as  an  issue 
Interfaces  are  assumed  to  be  so  natural 
as  to  make  training  urmecessary; 
naturairtess  is  also  assumed  to 
promote  skill  transfer  from  SE  to  real 
world  if  SE  is  being  used  as  traiiting  tool 


Table  12:  Comparison  of  Different  Design  Methods  for  telepresence 

(from  [Draper  et  al  99]) 


Sheridan  [Sheridan  92],  from  an  engineering  perspective,  describes  4  different  classes  of 
HMI  interface  systems  (at  the  left)  and  their  displays  (right)  as  shown  in  Figure  50. 


64 


d  PrBcognitivt  svtttm  an<t  dtsplav 

Figure  50:  Sheridan’s  4  classes  of  HMI  designs  (from  [Sheridan  92]) 

3.7.2  Display  Design  Considerations 

The  National  Research  Council  study  on  Tactical  Displays  for  Soldiers  [97]  provides  guidelines 
for  display  system  design  of  potential  use  in  TMR. 

1.  Design  should  minimize  itself  as  a  physical  barrier  to  acquiring  environmental  informa¬ 
tion  such  as  sight  and  sound. 

2.  It  should  enhance  sensory  input  only  when  required. 

3.  It  should  not  distract  attention. 

4.  It  should  minimize  cognitive  load  on  operator  by: 

•  Providing  fused  information  from  multiple  sources. 

•  Allow  for  easy  input  of  information. 

•  Minimize  memory  needs. 

•  Reduce  irrelevant  information. 

•  Simplify  information  presentation. 

•  Reduce  tasks. 

•  Present  information  in  properly  ordered  sequence  for  task. 

•  Provide  information  in  correct  format  (e.g.,  maps). 

•  Provide  suitable  cueing  and  attention  directing  mechanisms 

5.  Design  should  minimize  complexity  and  avoid  complex  automation. 

6.  Should  provide  new  capabilities  to  soldier. 


65 


7.  Should  allow  for  the  easy  sharing  of  information. 

[Draper  et  al  1991]  conducted  a  series  of  experiments  to  determine  when  and  if  to  use  stereo¬ 
scopic  television  displays  for  teleoperation.  There  results  showed  that  is  was  heavily  task  de¬ 
pendent  in  terms  of  performance  improvement.  They  contend  that  "it  is  necessary  to  use  an  eco¬ 
logical  perspective  when  considering  teleoperator  viewing  systems",  hence  it  is  crucial  to  under¬ 
stand  the  task-environment  in  which  the  robot(s)  resides.  So  whether  or  not  stereo  display  sys¬ 
tems  are  suitable  for  TMR  type  tasks  will  require  specific  task-sensitive  studies  to  be  performed. 

3. 7.3  Task  Analysis  Models  for  Design 

[Draper  et  al  99]  show  the  use  of  a  task  analysis  model  for  the  development  of  a  DOD  applica¬ 
tion  for  the  navy  (flight  deck  aircraft  servicing).  This  formal  approach  assists  in  the  design  of  an 
effective  system,  based  on  what  is  needed  for  what  tasks.  Summarizing  the  approach: 

1 .  Identification  of  tasks  and  task  elements. 

2.  Initial  screening  of  tasks  in  terms  of  their  impact  on  workload,  and  perceptual,  cogni¬ 
tive,  and  movement  requirements. 

3.  Development  of  task  network  models,  establishing  relationships  between  tasks. 

4.  Identification  of  task  requirements  for  each  task. 

5.  Identification  of  constraints  on  operation  (physical  and  operational)  to  allow  for  de¬ 
velopment  of  alternatives. 

6.  Identification  of  subtasks  (task  decomposition). 

7.  Identification  of  replacement  and  alternatives  (where  robotics  can  best  fit). 

8.  Evaluation  of  gains. 

This  process  is  iterative.  The  study  cited  above  can  serve  as  a  potential  exemplar  for  the  devel¬ 
opment  of  a  similar  TMR-related  study.  One  of  the  key  outcomes  would  be  a  sound  justification 
for  the  use  of  robotics  technology  in  certain  TMR-related  task  areas. 

3.7.4  Alarm  Design 

[Riera  et  al  95]  discuss  the  role  of  alarms  in  supervisory  processes  when  operators  are  in  a 
removed  supervisory  room.  This  may  only  extend  in  some  ways  to  the  work  in  TMR.  Two  cate¬ 
gories  of  alarms  are  cited:  not-programmed  alarms  that  arise  from  external  events  (sensors  for 
intrusion,  overheating,  etc.)  and  programmed  alarms  defined  at  time  of  design  (threshold  excep¬ 
tions,  etc.).  These  programmed  alarms  are  of  2  types:  breakdown  (failure  of  physical  compo¬ 
nents)  and  process  (abnormal  condition  or  behavior).  They  provide  an  inductive  method  from 
dependability  science  as  a  basis  for  alarm  design,  which  involves  a  functional  analysis  of  the  sys¬ 
tem. 

[Kuchar  and  Hansman  95]  describe  methods  for  evaluating  the  performance  of  these  alert¬ 
ing  systems.  Figure  51  shows  how  alerting  systems  relate  to  both  the  operator  and  environment. 
This  approach  incorporates  a  human  response  model  in  its  design. 
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Figure  51:  Alerting  System  Model  (from  [Kuchar  and  Hansman  95]) 


3.7.5  Teleoperator  interface  design 

Research  at  SPAWAR  [Gilbreath  et  al  2000]  used  several  techniques  to  reduce  cognitive 
load  on  the  user  of  a  teleoperated  Mobile  Robot  (Robart  n).  Human-centered  mapping  is  one 
such  strategy  where  a  human  guides  map  building  and  exploration.  The  operator  guides  the  robot 
into  an  area,  using  natural  commands  (e.g.,  follow  the  wall  on  the  left").  This  disambiguates  sen¬ 
sor  data,  as  the  robot  can  use  the  human  input  for  labeling  purposes.  This  is  also  one  of  the  very 
few  systems  that  document  weapon  control  (non-lethal)  for  a  mobile  robot.  Unfortunately  no  user 
studies  were  conducted  to  validate  the  hypothesis  that  these  techniques  are  of  direct  benefit  to  the 
end-users. 

Additional  SPAWAR  work  studied  teleoperation  in  sewers  and  tunnels  [Laird  et  al  2000], 
which  clearly  are  TMR  relevant  tasks.  In  this  instance,  user  studies  were  conducted  on  a  reflexive 
teleoperated  interface.  It  was  said  that  the  interface  provided  more  sophistication  than  required  by 
the  soldier,  the  soldiers  preferring  a  purely  teleoperated  strategy,  interestingly.  More  automation 
is  clearly  not  always  better.  A  heads-up  display  was  useful  within  the  tunnels  but  not  outside  due 
to  glare.  Another  complaint  was  the  inability  to  view  the  video  by  more  than  one  soldier  at  a 
time.  Lack  of  a  zoom  feature  on  the  camera  was  also  a  complaint.  A  second  generation  OCU  was 
created  but  has  not  yet  been  field-tested.  In  any  case,  the  soldiers  preferred  slow  and  steady  ex¬ 
ploration  rather  than  faster  semi- autonomous  operation  for  this  task. 

[Murphy  and  Rogers  96]  describe  TeleSFX/TeleVIA,  a  system  that  provides  a  method  for 
semi-autonomous  control  of  robots  that  incorporates  user  modeling.  They  introduce  the  notion  of 
tele-assistance  where  the  operator  only  provides  support  for  the  robot,  not  control.  Here  an  inter¬ 
vening  computational  agent  between  the  human  and  the  operator  facilitates  the  communication 
and  control  between  them.  The  system  uses  focus  of  attention  and  hypotheses  panels  to  help  di¬ 
rect  the  operator’s  attention  and  goals.  The  examples  presented  are  more  for  error  detection  and 
diagnosis  rather  than  for  navigation,  helping  the  operator  to  determine  what  to  do  in  the  light  of 
sensory  malfunction.  Having  the  intervening  agent,  between  robot  and  operator,  digest  and  proc¬ 
ess  the  information  prior  to  requiring  operator  intervention  minimizes  communication.  The  ex- 
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tension  of  this  work  to  a  broader  class  of  failure  diagnosis  is  clear  and  could  make  a  useful  sup¬ 
port  agent  for  this  single  aspect  of  TMR  mission  conduct  (sensor  management). 


3.8  Operator  Selection 

Regarding  operator  selection,  [Johnsen  and  Corliss  1967]  state  that  "The  operators  [for  telerobot¬ 
ics]  should  be  selected  with  the  same  care  as  for  jet  pilots."  In  traditional  teleoperation  domains, 
they  refer  to  the  following  essential  capabilities  (p.  65): 

•  Excellent  depth  perception 

•  Superior  eye-hand  coordination 

•  Display  stability  and  resourcefulness  in  hostile  environments 

•  Good  physical  condition 

This  assessment  culminates  in  the  statement  that  "Good  manipulator  operators  are  hard  to  find". 
Clearly  TMR  then  should  pay  attention  to  the  types  of  personnel  dedicated  to  these  tasks. 

3.8.1  Individual  Differences 

Ackerman  (currently  at  Georgia  Tech)  has  addressed  the  question  of  identifying  individual 
differences  between  operators  for  air-traffic  control  and  other  complex  skill  acquisition  tasks. 
This  can  aid  in  determining  who  may  qualify  as  the  best  TMR  operators.  In  one  study,  he  ad¬ 
dresses  ability  testing  methods  for  general,  reasoning,  spatial,  perceptual  speed,  and  percep¬ 
tual/psychomotor  abilities  for  assessing  terminal  radar  approach  tasks.  He  has  developed  a  theory 
of  ability  determinants  (Figure  52)  that  relates  skill  acquisition  to  different  people.  It  is  broken 
into  three  phases:  cognitive,  psychomotor,  and  autonomous.  While  beyond  the  scope  of  this  re¬ 
port  to  discuss  the  underlying  theoretical  implications,  it  is  of  note  that  this  theory  can  be  used  to 
design  tests  that  can  be  applied  to  tasks  to  differentiate  who  might  be  the  best  operators  for  multi¬ 
robot  control  tasks.  Some  generic  conclusions  on  the  air  traffic  controller  test  that  were  gender 
based,  include  that  men  perform  better  on  spatial  ability  tasks,  while  women  outperform  men  on 
perceptual  speed  tasks  [Ackerman  92].  The  key  for  TMR  is  understanding  what  tasks  are  in¬ 
volved.  The  analysis,  if  desired  could  be  extended  to  other  subject  dimensions  as  well:  education, 
age,  etc. 


68 


Ability  Structure 


Skill  Acquisition 


Genitive 


Associative 
Phase 


Autonomous 

Phase 


Figure  52:  Ackerman’s  theory  of  Ability  Determinants  (from  [Ackerman  92]) 

Ackerman’s  initial  work  led  to  a  battery  of  tests  that  are  used  to  evaluate  potential  air  traffic  con¬ 
trollers  [Ackerman  and  Kanfer  93].  A  wide  range  of  tests  (12),  including  things  as  seemingly 
simple  as  paper  folding,  letter/number  substitution,  dial  reading,  and  necessary  facts,  to  other 
tests  involving  spatial  memory,  problem  solving  and  verbal  tests  were  used  to  develop  a  compos¬ 
ite  of  a  potential  controller.  It  is  entirely  possible  that  a  similar  set  of  attributes  might  also  apply 
to  TMR-type  tasks,  but  a  formal  task  analysis  would  need  to  be  applied.  This  same  battery  of 
tests  however  may  be  useful,  in  differentiating  just  who  might  be  the  best  controller  given  an  ac¬ 
curate  mapping  of  task  criteria  onto  operator  performance.  Even  more  recently,  he  [Ackerman 
99]  has  extended  this  to  touch-panel  testing,  which  potentially  can  provide  an  even  more  realistic 
interface  assessment  in  conjunction  with  an  operator’s  potential  for  performing  in  a  TMR  envi¬ 
ronment. 

[Duncan  et  al  93]  explored  attention  in  the  context  of  distraction.  The  test  involved  driving  a  car 
in  an  urban  environment.  This  provides  a  measure  for  how  easily  distracted  potential  operators 
are,  and  can  serve  as  measure  of  individual  operator  ability  for  TMR  related  tasks. 

3.8.2  Skills  needed 

Air  traffic  controllers  (ATC),  who  are  charged  with  spatial  management  tasks  TMR  have  been 
singled  out  as  having  much  potentially  in  common  with  TMR  operators  controlling  multiple  ve¬ 
hicles.  [Wickens  et  al  98]  describe  9  ability  categories  that  an  ATC  needs: 

1 .  Spatial  reasoning  ability 

2.  Verbal  reasoning  ability 

3.  Numerical  reasoning  ability 

4.  Perceptual  speed  factor  for  coding 

5.  Selective  attention 
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6.  Short-term  memory 

7.  Long-term  memory 

8.  Time-sharing 

9.  Manual  dexterity. 


Verbal  reasoning  perhaps  is  the  only  one  that  might  be  extraneous  to  TMR.  Much  could  be 
learned  at  looking  in  detail  at  training  procedures  for  ATCs  and  seeing  how  similar  processes 
could  be  applied  in  this  program.  Tables  13-15  illustrate  task  sets  for  and  attributes  of  Air  traffic 
controllers.  Similar  tabular  analysis  could  be  applied  to  TMR  robot  unit  tasks 


Ta.sk  Set 

Cognitivc/Sensory  Attributes 

Number  of  Tasks 

Entry 

Coding 

134 

Receipt 

Movement  detection 

3 

.Spatial  scanning 

34 

Filtering 

43 

Image/pattern  recognition 

42 

Decoding 

157 

Analysis 

Visualization 

7 

Short-term  memory 

35 

Lx)ng-tcrm  memory 

lU 

Deductive  rea.soning 

107 

Inductive  rea.suning 

16 

Probabilistic  reasoning 

43 

Prioritization 

23 

Communication 

Verbal  filtering 

132 

Table  13:  Task  set/Cognitive  attributes  for  ATCs  (from  [Wickens  et  al  98]). 
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AUributes 

Examples 

Coding 

Enter  information  into  the  maintenance  log 

Movement  detection 

Listen  for  alarm  printouts 

Spatial  scanning 

Observe  status  panels  for  status  data 

Filtering 

Identify  significant  status  data  on  status  panel 

Pattern  recognition 

Form  mental  picture  of  facility  status 

Decoding 

Read  a  facility  configuration  display  screen 

Visualiration 

Determine  operations  impacts  from  weather  picture 

Short-term  memory 

Remember  status  information  to  record  in  log 

Long-term  memory 

Remember  procedures 

Deductive  reasoning 

Determine  that  facility  data  arc  questionable 

Inductive  reasoning 

Hstimate  impact  from  historical  trend  data 

Probabilistic  reasoning 

Evaluate  the  nature  of  a  degradation 

Prioritization 

Establish  order  for  restoring  equipment 

Verbal  filtering 

Identify  relevant  verbal  information 

Table  14:  Example  ATC  cognitive  attributes  (from  [Wickens  et  al  98]). 

Cognitive  Funttions 


(Higher  To  Lower) 

Cognitivc/Sensory  Attributes 

Number  ol  Tasks 

Plan/resolvc 

Prioritizing 

li 

Predict  longer  term 

Inductive  reasoning 

lb 

Compare,  predict 
shorter  term 

Deductive  reasoning,  pattern 
recognition,  probabilistic 
reasoning,  visualization 

199 

Transmit  information 

Coding,  decoding,  verbal  filtering 

42.1 

Remember 

Short-term  memory,  long-term 
memory 

4S 

Identify 

Filtering,  movement  detection, 
spatial  scanning 

80 

Table  15:  Hierarchy  of  attributes  of  ATCs  (from  [Wickens  et  al  98]) 


3.8.3  Tunnel  Vision  as  basis  for  selection 

Tunnel  vision  is  defined  by  Boer  [Boer  1995]  as  a  narrowing  of  attention  by  concentrating  on  a 
part  of  system  where  the  operator  loses  the  overview.  Narrowing  of  attention  occurs  as  mental 
workload  increases.  At  high  levels  of  workload  task-relevant  stimuli  are  excluded.  This  occurs  in 
situations  for  example  when  the  operator  is  handling  fault  management.  In  certain  cases,  tunnel 
vision  also  occurs  due  to  waiting  for  information.  Boer  suggests  that  it  is  possible  to  assess  the 
proneness  of  operators  through  tests,  and  where  necessary  provide  remedial  training  for  "weak" 
operators.  Of  course  if  this  is  a  criteria  for  selection,  these  "weak"  operators  who  are  prone  to 
tunnel  vision  can  be  excluded  from  the  population  of  potential  TMR  operators. 
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3.9  Trainins  operators 


Perhaps  the  best  results  regarding  guidelines  for  training  teleoperators  and  operators  of  autono¬ 
mous  systems  is  contained  in  the  Final  Report  of  a  Workshop  sponsored  by  DOE  entitled 
"Guidelines  for  the  Development  of  Training  Programs  for  the  Operation  and  Maintenance  of 
Robotic  Equipment"  [Byrd  and  Duke  98].  This  report,  provided  by  John  Draper  from  ORNE,  is 
highly  relevant  to  TMR  as  it  describes  in  details  the  duties,  tasks  and  elements  for  teleoperators, 
advanced  teleoperators,  and  autonomous  systems.  These  are  mapped  onto  the  knowledge,  skills, 
and  abilities  sets  that  individuals  either  need  to  possess  or  be  trained  to  possess.  Relevant  por¬ 
tions  of  this  report  are  reproduced  in  Appendix  A.  These  can  be  easily  mapped  onto  Soldier  train¬ 
ing  publications  (STP)  and  Training  Support  Packages  (TSPs)  as  used  by  the  DOD  in  general. 
New  Military  Occupational  Specialties  (MOS)  can  be  created  for  telerobotic  operators,  as  the 
skill  sets  in  this  DOE  study  are  clearly  enumerated  and  many  of  which  would  be  highly  relevant 
to  TMR.  This  report  can  provide  guidance  in  the  formulation  of  DOD  mission  training  plans  as 
well.  STPs,  TSPs,  and  mission  training  plans  for  typical  infantry  tasks  can  be  found  at  the  home 
page  for  the  Army’s  Training  and  Doctrine  Digital  Eibrary  (http://155.217.58.58/atdls.htm).  A 
suitable  follow-on  task  for  TMR  would  be  to  attempt  to  develop  a  TSP  and  STP  for  specific 
TMR  relevant  MOSs,  e.g.,  robotic  operation  and  maintenance,  based  on  the  datapool  provided  in 
the  DOE  report. 

[Chappell  and  Mitchell]  point  out  that  well  trained  novices  does  not  result  in  expert  performance. 
Expert  operators  anticipate  patterns  of  normal  operation  and  quickly  recognize  prototypical  fail¬ 
ure  modes,  while  exhibiting  confidence  in  their  ability.  Various  techniques  can  be  used  to  train 
novices  including  computer-based  instruction,  low-fidelity,  high-fidelity  and  high-fidelity  with 
motion  based  simulators.  Often  the  compression  of  large  volumes  of  materials  results  in  stress  in 
the  trainees.  The  authors  recommend  the  introduction  of  intelligent  tutoring  systems  to  bridge  the 
gap  from  trained  novice  to  expert.  These  systems  are  tailored  to  the  individual  student’s  needs. 

Gopher  [Gopher  93]  studied  issues  regarding  the  role  of  attention  control  and  how  to  train  opera¬ 
tors  to  perform  better.  In  particular,  he  was  concerned  with  strategic  control  of  attention,  i.e.,  do 
people  know  how  well  they  invest  attention  and  execute  related  attentional  strategies.  Successful 
examples  include: 

•  Eocusing  attention  (voluntary  selection). 

•  Dividing  attention  (coping  with  concurrently  presented  tasks). 

•  Switching  attention  (voluntary  movement  of  control  from  one  task  to  another). 

•  Investing  graded  levels  of  effort  (dividing  attention  over  tasks  of  differing  priorities). 

Problems  that  occur  in  people  in  managing  attention  include: 

•  Eailure  to  protect  primary- task  performance. 

•  Dissociation  between  subjective  estimates  and  performance  measures  of  workload. 

•  Eack  of  adequate  feedback. 

•  Attention  capture  by  automatic  components  (subjects  needed  to  be  trained  to  be  able  to 
relax  and  release  attention). 

•  Ability  to  release  resources  voluntarily. 
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Gopher  developed  a  series  of  tests  [Gopher  93]  that  are  geared  to  enhancing  an  operator’s  ability 
to  manage  attention  more  effectively,  a  crucial  role  for  control  of  multirobot  missions.  In  particu¬ 
lar  he  used  videogame  scenarios  to  develop  these  skills,  which  might  account  for  the  intuition 
that  those  who  are  good  at  video  games  might  also  be  good  for  TMR  related  tasks. 

[Zachary  et  al  98]  describe  a  training  environment  for  tactical  teams.  It  provides  performance  as¬ 
sessment,  cognitive  diagnosis,  and  instructor  support  in  addition  to  simulation  facilities.  The  task 
domain  is  for  the  Air  Defense  Team  component  of  the  Navy’s  AEGIS  command  and  control  sys¬ 
tem.  The  AETS  training  system  embeds  cognitive  models  of  each  of  the  trainees.  The  value  of 
embedding  models  is  predicting  actions  and  diagnosing  faults.  Unfortunately  no  data  was  avail¬ 
able  regarding  the  efficacy  of  this  method. 

Other  new  approaches  to  training  [Wickens  et  al  98]  include  the  use  of  embedded  training  where 
the  training  resides  within  the  actual  operational  system.  These  embedded  trainers: 

1.  Require  operators  to  perform  tasks  in  response  to  simulated  inputs 

2.  Present  realistic  scenarios  including  downgraded  operational  modes 

3.  Be  interactive  in  assessing  the  operators  ability 

4.  Record  performance  and  provide  feedback. 


3.10  Cosnitive  Modelins 

[Kirlik  et  al  1993]  studied  human  performance  in  a  simulated  task  related  to  some  potential 
TMR  applications.  In  particular  they  looked  at  scout  aircraft  operating  in  dynamic  and  uncertain 
environments  that  could  be  controlled  either  manually  via  joysticks  or  through  the  designation  of 
waypoints  via  a  map  interface.  One  and  two  person  crews  were  studied.  The  scout  craft  in  which 
the  operator  was  notionally  housed  controlled  4  semiautonomous  friendly  craft  with  the  goal  of 
finding  desirable  target  objects  within  the  world.  A  large  number  of  enemy  craft  were  also  pre¬ 
sent.  While  the  task  itself  was  somewhat  contrived,  it  did  give  rise  to  an  affordance-based  model 
of  cognition  that  was  used  to  account  for  the  human  performance  in  the  task.  The  outcome  of  the 
work  in  regards  to  user  interfaces  is  that  automated  assistance  could  be  designed  based  on  the 
user  model  and  that  highly  effective  graphical  representation  could  be  created  based  on  the  affor- 
dance  that  the  system  needs  to  convey  to  the  user,  rather  than  on  some  arbitrary  available  repre¬ 
sentation.  Similar  modeling  processes  could  be  applied  to  TMR  operators,  that  would  result  in 
analogous  models  to  what  is  depicted  in  the  Eigure  53  that  resulted  for  this  particular  task. 
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Figure  53:  Kirlik’s  Model  of  Scout  Search  path  planning  (from  [Kirlik  et  al  93]) 

[Callantine  and  Mitchell]  have  studied  how  operators  select  and  use  various  automation 
modes  to  control  complex  dynamic  systems.  This  methodology  is  potentially  applicable  to  TMR- 
type  tasks  for  tracking  just  how  operators  perform  their  tasks.  The  cognitive  model  on  which  it  is 
based  is  called  OFM-ACM:  Operator  Function  Model  for  Systems  with  Automatic  Control. 
While  this  system  has  been  tested  in  flight  cockpits  for  Boeing  757/767  simulators  it  may  be  ex¬ 
tendible  to  other  domains  such  as  TMR  as  well.  No  explanations  for  actions  are  provided  just 
tracking  and  organization  of  the  user  data  itself. 


3.11  Summary  Recommendations  -  Human  Factors 

The  assumptions  underlying  the  recommendations  in  this  section  are  based  upon  discussions 
with  LTC  John  Blitch,  DARPA  Program  Manager  for  the  Tactical  Mobile  Robotics  Program.  We 
are  considering  a  system  where  a  field  operator  may  be  capable  of  controlling  up  to  4  TMR  units 
simultaneously.  These  may  be  coupled  with  multiple  operators,  each  capable  of  managing  similar 
units.  At  issue  is  what  should  the  interface  be  like  in  terms  of  managing  load,  what  types  of  skills 
are  necessary  for  such  an  operator,  how  could  training  be  most  effectively  conducted,  and  how 
could  we  evaluate  performance.  The  following  recommendations  speak  directly  to  these  issues  in 
that  context. 

It  must  also  be  stated  as  there  are  no  studies  that  specifically  address  the  human  factors  for 
this  task  domain,  the  basis  for  these  recommendations  is  from  extrapolation  of  studies  that  typi¬ 
cally  consider  only  one  or  a  few  of  the  related  issues.  As  such  these  recommendations  are  specu¬ 
lative  and  must  be  confirmed  through  formal  studies  to  truly  be  considered  well  founded.  They 
do  provide  a  basis  for  generating  hypotheses  for  subsequent  formal  testing  in  TMR  scenarios. 
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which  is  a  strong  recommendation  for  the  TMR  program.  For  many  of  the  recommendations  be¬ 
low,  the  reference  study  on  which  it  is  based  is  cited  alongside  the  recommendation  itself. 


1 .  It  is  clear  that  straight  teleoperation  will  be  inadequate  for  such  a  complex  task,  due 
to  the  cognitive  difficulties  in  managing  multiple  robotic  units,  with  the  additional 
complexity  of  conducting  these  operations  in  stressful  environments.  Low-level  su¬ 
pervisory  control  is  even  questionable  in  terms  of  providing  the  necessary  operator 
support.  Thus  specialized  high-level  supervisory  control  with  the  ability  to  treat  the 
robots  as  a  unit,  or  fully  automatic  (autonomous)  control,  with  alert-based  operator 
intervention,  is  likely  the  best  course  [Sheridan  93]. 

2.  Telepresence  is  likely  not  a  viable  solution,  due  to  the  need  for  the  military  operator 
to  maintain  an  awareness  of  his  surroundings  as  a  defensive  posture.  Thus  situational 
awareness  must  be  twofold:  local  to  the  operator  as  well  as  remote  from  the  distrib¬ 
uted  robotic  team. 

3.  Regarding  specific  sensory  cues  for  inclusion  in  an  OCU  for  TMR: 

a.  Vibratory  information  can  be  useful  to  provide  feedback  on  state  achieve¬ 
ment,  or  where  it  may  increase  the  subjective  feel  of  the  operator  to  the  ro¬ 
bot’s  environment  as  in  the  presence  of  an  enemy  [Kontarinis  and  Howe]. 

b.  Mixed  haptic/visual  feedback  has  been  demonstrated  to  provide  enhanced 
control  over  purely  visual  feedback  systems.  Specific  data  is  provided  in  Sec¬ 
tion  3.2.3  for  guidance  in  the  design  of  such  a  system  based  on  the  limitations 
of  human  spatiotemporal  response. 

c.  Auditory  presentation  (given  a  sufficiently  quiet  environment)  is  valuable  for 
short  simple  messages  that  do  not  need  to  be  remembered  and  that  are  in¬ 
volved  with  current  events.  They  can  be  of  particular  value  if  the  operator  is 
on  the  move.  Stereo  and  other  forms  of  auditory  presentation  can  provide  3D 
information  regarding  the  source  of  an  event  (e.g.,  presence  of  target)  [Burdea 
96]. 

d.  Motion  sickness  in  the  operator  is  something  that  needs  to  be  considered  for 
any  form  of  heads-up  or  helmet-mounted  display  when  the  operator  is  ambu¬ 
lating  or  driving.  Coupling  the  visual  feedback  from  the  robot  with  the  mo¬ 
tion  of  the  observer  seems  one  of  the  better  approaches  to  avoid  this  effect. 

4.  Regarding  display  characteristics: 

a.  If  the  operator  will  be  on  the  move  and  subject  to  vibratory  stresses,  bar 
graphs  are  recommended  over  polygonal  representations  as  graphics,  as  these 
have  been  shown  to  be  more  accurately  interpreted  [Desombre  et  al  95]  under 
these  conditions. 

b.  Operator  stimuli  should  be  presented  in  an  easy  to  discriminate  manner  that 
reduces  visual  search  or  scanning  time  [Sheridan  and  Ferrell  74]. 
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c.  For  monitoring  tasks,  where  boredom  or  tedium  may  be  a  factor,  means  by 
which  to  keep  the  operator  engaged  with  the  TMR  robots  is  important  to 
avoid  the  so-called  vigilance  decrement.  Artificial  tasks  can  be  added  for  this 
purpose  [Thurman  and  Mitchell95].  As  TMR  is  likely  to  have  robots  that 
possess  a  high  level  of  autonomy,  it  is  important  that  the  operator  not  be 
lulled  into  a  false  sense  of  security. 

d.  Use  of  scene-linked  symbology  for  heads-up  displays,  superimposed  on  re¬ 
turning  video  imagery  is  strongly  recommended  to  attain  real-time  response 
for  events  that  need  such  attention.  The  use  of  perspective  (distance- 
dependent  symbol  size),  differential  motion,  and  natural  coloration  can  also 
reduce  attentional  tunneling  in  the  operator[Foyle  et  al,  McCann  et  al]. 

e.  A  helmet-mounted  display  should  not  occlude  the  soldier’s  view  in  the  field. 
Wrist-mounted  or  hand-held  displays  are  recommended  as  an  alternative  if 
the  HMD  cannot  be  so  designed  [National  Research  Council]. 

f.  Egocentric  coordinate  systems  for  displays  are  preferred  over  absolute  geo¬ 
graphic  ones  [McCann  et  al]  to  reduce  the  need  for  mental  transformations  in 
the  operator’s  head.  How  this  can  be  accomplished  with  groups  consisting  of 
multiple  robots  is  harder  to  determine  at  this  time,  and  warrants  additional  in¬ 
vestigation. 

g.  The  OCU  must  be  designed  recognizing  that  it  will  be  used  in  a  stressful  en¬ 
vironment,  incorporating  the  specific  stress  reduction  guidelines  presented  in 
Section  3.6.4  [National  Research  Council].  Also  incorporate  the  other  design 
guidelines  from  this  NRC  study  in  the  OCU’s  display  design  (Section  3.7.2). 

h.  Employ  a  tested  design  methodology  for  OCU  design,  rather  than  an  ad  hoc 
approach.  Candidates  from  Section  3.7.1  and  3.7.3  include  [Corliss  et  al  68], 
[Johannsen  97]  and  [Draper  et  al  99].  A  mission-oriented  approach  as  advo¬ 
cated  by  [Draper  93]  is  recommended. 

5.  Regarding  robot  design  characteristics: 

a.  Camera  mounting  position  on  a  TMR  if  used  for  visual  feedback  control  can 
have  significant  impact  on  operator  performance  [Eevison  and  Baron  97]. 
Human  factors  studies  should  be  undertaken  to  evaluate  just  what  the  effects 
are  of  low-mounted  ground  hugging  cameras  on  operator  performance.  No 
such  study  is  in  existence  to  the  best  of  our  knowledge. 

6.  Regarding  Operator  Performance: 

a.  There  are  fundamental  concerns  as  to  whether  humans  are  capable  of  manag¬ 
ing  complex  distributed  sensing  and  actuation  environments  at  all  [Murray 
95].  Serious  studies  should  be  undertaken  early  in  TMR,  to  evaluate  if  opera¬ 
tors  are  potentially  capable  of  managing  as  complex  a  task  as  envisioned  for  4 
robot  units.  To  a  degree,  this  will  determine  the  requisite  level  of  autonomy 
for  the  robots,  if  the  task  would  otherwise  require  significant  operator  interac¬ 
tion  (and  not  simply  monitoring). 
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b.  It  does  not  always  follow  that  more  automation  improves  operator  perform¬ 
ance  [Wei  et  al  95].  Judicious  choice  of  automation  techniques  should  be 
made  in  the  design  of  TMR,  avoiding  the  use  of  automation  for  automation’s 
sake.  This  is  further  confirmed  for  TMR- like  tasks  for  single  robots  operating 
within  sewers  [Laird  et  al  2000] . 

c.  Human  error  occurs  most  frequently  during  intervention  [Sheridan  93].  Spe¬ 
cial  care  should  be  engineered  into  the  system  to  reduce  this  possibility. 

d.  Appropriate  levels  of  trust  must  be  developed  between  the  operator  and  the 
robots,  somewhere  between  mistrust  and  overtrust  [Wickens  et  al  98].  Section 

3.6.2  cites  the  relevant  criteria  which  must  be  addressed  in  design  and  train¬ 
ing. 

e.  Other  human  errors  to  which  attention  must  be  paid  include: 

i.  Mode  error  and  ambiguity  [Dezani  et  al  98] 

ii.  Dual-task  interference,  a  potentially  very  serious  source  of  error  for 
controlling  groups  of  robots  [Van  Selst]. 

iii.  Misalignment  of  operator’s  mental  model  of  the  system  with  the  actual 
system  [Palmer] .  This  can  be  partially  addressed  through  training. 

7.  Regarding  operator  selection: 

a.  Different  individuals  have  different  abilities.  Care  should  be  taken  in  operator 
choice.  It  appears  very  unlikely  that  all  soldiers  will  be  properly  suited  for 
TMR  tasks  involving  teams  of  robots,  even  with  suitable  training.  Potentially, 
a  good  TMR  operator  may  be  far  and  few  between. 

b.  Test  batteries  exist  that  can  be  used  to  help  determine  which  individuals  are 
likely  to  be  best  suited  for  TMR  operations.  It  is  recommended  that  air- traffic 
controller  tests  be  considered  as  a  point  of  departure  for  such  evaluations 
[Ackerman  and  Kanfer  93].  They  appear  easy  to  administer  and  likely  will 
need  to  be  supplemented  with  additional  functionality.  A  first  cut  based  on 
ATC  operators  regarding  the  skills  needed  for  TMR  is  presented  in  Section 

3.8.2  [Wickens  et  al  98]. 

c.  Specialized  attentional  testing  to  avoid  including  operators  who  are  prone  to 
attentional  tunnel  vision  is  also  recommended  [Boer  95]. 

8.  Regarding  operator  training: 

a.  In  order  to  avoid  adverse  impact  on  response  time,  training  should  focus  on 
familiarizing  operators  with  relevant  TMR  stimuli  and  their  presentation:  to 
reduce  their  unexpectedness  [Sheridan  and  Ferrell  74]. 

b.  Appendix  A  contains  guidelines  for  the  training  for  the  operators  for  robotic 
equipment  developed  by  the  DOE.  These  guidelines  should  be  seriously  con¬ 
sidered  and  expanded  appropriately  for  TMR-related  tasks  [Byrd  and  Duke 
98]. 
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c.  The  use  of  intelligent  tutoring  systems  integrated  directly  into  the  OCU  ap¬ 
pears  to  simplify  in-situ  training  of  operators  [Chappell  and  Mitchell]. 

d.  Tests  to  improve  the  ability  of  an  operator  to  manage  attention  are  available, 
which  is  crucial  for  a  multirobot  environment  [Gopher  93].  These  should  be 
considered  for  applicability  to  TMR. 

e.  Training  environments  can  also  be  of  value  (which  is  well-known)  and  per¬ 
haps  TMR  should  consider  investing  in  such  an  embedded  simulation  system 
[Wickens  et  al  98]. 

9.  Evaluation: 

a.  Definition  of  metrics  for  task  and  operator  performance  must  be  defined 
clearly  and  early  in  the  design  process.  Traditional  metrics  include  task  per¬ 
formance  time  and  success  ratio,  but  measurements  of  operator  stress,  fatigue, 
and  workload  management  are  certainly  necessary  as  well  for  TMR. 

b.  The  workload  measurement  rating  system  (Figure  45)  or  the  NASA  TLX 
workload  index  [Draper  and  Blair  96]  appear  to  be  suitable  candidates  for  the 
basis  of  quantifying  operator  performance  for  TMR  tasks. 

c.  Alarm-based  operator  performance  can  be  assessed  using  methods  similar  to 
those  developed  by  [Kuchar  and  Hansman  95]  (Section  3.7.4). 
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Appendix  A.  Operator  trainins 

This  material  is  reproduced  from  Byrd,  J.,  and  Duke,  E.,  (Eds.)  "Einal  Report:  Guidelines  for  the 
Development  of  Training  Programs  for  the  Operation  and  Maintenance  of  Robotic  equipment", 
Morgantown,  WV,  U.S.  Department  of  Energy,  Eederal  Energy  Technology  Center, 
pp.  15,19,23,e-3-47 
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Appendix  B.  Human  Factors  Considerations  and  Display  Systems 

This  material  is  reproduced  from  National  Research  Council,  Tactical  Display  for  Soldiers 
Human  Factors  Considerations,  National  Academy  Press,  Washington,  D.C.,  1997.  pp.  70-87. 
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